Run VH Sandbox Example

Overview

Shows how Virtual Humans use a flexible set of backend solutions to demonstrate conversational behavior.

Leverages RIDE’s generic C# interfaces for Speech Recognition (ASR), Natural Language Processing (NLP), Text-To-Speech (TTS) and Non-verbal Behavior Generation (NVBG). Optionally includes sentiment analysis and entity analysis for NLP and Sensing (computer vision) with support for emotion and landmark recognition.

Initially supported services:

Supplemental Virtual Human Conversational Behavior:

• Gaze, Saccade
• Listening
• Emotion Mirroring

Review the Supported Services by Character and Platform section for which combinations of the above are currently expected to function.

Additionally, this scenario has the capability of utilizing custom user created backend implementations for any of the above systems. Requires knowledge of RIDE’s generic C# interfaces.

How to Use

Run the scene and use the sandbox menu to control the scenario parameters.

Environment Tab 

• Choose between six flat background images, including USC ICT logo, 3D lobby scene, and landscape photos.
• Three locations within the 3D SeaView environment.

VH Config Tab

• Create VH: In this sub-tab, you can configure the backends you want to test before creating the Virtual Human by selecting the Create button.
  • Character ID is prepopulated for each VH and is customizable; note, each VH must have a unique name.
  • TTS Voice list is based on the selected TTS backend.
  • Select Compare & Contrast to randomize the parameters prior to creation.
  • Virtual Humans are created and placed within player FOV.

• Edit VH:  In this sub-tab, you can swap the backends for already created Virtual Humans.
  • VH model template cannot be modified after creation.
  • VH position and rotation can be modified here.
  • Select between instanced VH characters from the dropdown. 
  • Delete problematic or unwanted VH characters.

• Save/Load VH: In this sub-tab, you can save and load configured Virtual Humans to/from local storage.
  • Saved VH configurations are identified by the VH name.
  • VH configurations are saved as JSON. If in-scene backend objects have their name changed, the previously saved configuration will not load properly.
  • If you load a VH configuration for an already created or loaded VH, the existing VH config will be overwritten by the loaded configuration.

Conversation Tab

• This tab acts as conversation analysis/debugging.
• Tracks the current VH the user is having a conversation with.
• Has 2 sub-tabs, Analysis and Debug Info, for more in-depth information about the current conversation.

User Input

• Text Input: Use the designated input field and the Submit button to send a query to the current conversational VH.
• Audio Input: Press and hold the Push-To-Talk button and begin speaking.
  • Input field will display detected partial speech (if supported by the current ASR backend).
  • Query will be sent either once you release the push-to-talk button, or automatically if complete speech is detected, based on the current ASR system.

Toggle Webcam

• Turns on the webcam and begins sensing for the backend of the current conversational VH.
• Requires a created VH with an assigned Sensing backend.
• Webcam overlay displays the following information:
  • Detected face rectangle
  • Detected landmarks
  • Detected head orientation
  • Detected emotion

Key bindings:

Scene Location & Name

Assets/Scenes/VHSandbox/VHSandbox.unity

Setup Requirements 

The main script name and location: Assets/Scripts/VHSandboxExample.cs.

Below are the steps to add a custom VH backend implementation to the sandbox:

1. Create your custom backend script, that implements one of the following Ride C# interfaces:
   1. ASR – ISpeechRecognitionSystem
   2. NLP – INLPQnASystem
   3. TTS – ILipsyncedTextToSpeechSystem
   4. NVBG – INonverbalGeneratorSystem
   5. Sensing – ISensingSystem
2. Create a Unity gameobject with a user-friendly name and attach the created script.
   1. The Gameobject should be placed in the scene under the  “VH Backends” object, and should not have the same name as another backend gameobject.
3. To confirm the previous steps have been completed correctly, play the scene and see if the created gameobject name is displayed under the relevant backend dropdown.

Supported Services by Character and Platform

Note: * denotes a known issue.