Disclaimer: Preview version; content and features subject to change.
This tutorial is geared towards any researcher and developer who wants to new REST web services to VHToolkit 2.0.
This tutorial shows how developers can add additional NLP web services to RIDE. We’ll use Anthropic Claude as an example.
Background:
The scene consists of three objects: RideSystems,, a Camera, and the ExampleLLM gameObject. The initial game view will look like this:
The ExampleLLM child object hierarchy is as follows:
The ExampleLLM componen has the two main objects of interest that exist in the scene: InpQuestion, to register the user input, and the ProviderPanelParent, to track the provider panels that exist in the scene.
Existing in the hierarchy is the OpenAI GPT – 4 object under ExampleLLM -> ProviderPanelParent -> OpenAI GPT – 4. This is a PanelLLM prefab (Assets -> Ride (local) -> Prefabs -> UI -> PanelLLM.prefab) that has been modified to function as a UI panel for GPT4.
The PanelLLM.prefab hierarchy:
In addition to storing references to the RideTextTMPro components for use in updating the text of the relative sections, this component also allows for the customization of the colors of both the user input and the generated responses as they are displayed in the conversation text box.
Finally, the provider system is added to allow for the referencing of the NLP system that the panel will be showcasing the responses of.
In order to test a new provider, a new panel is needed to show its responses. To add a new panel for the Anthropic – Claude provider, open the ExampleNLP Prefab, and add the PanelLLM prefab (Assets -> Ride (local) -> Prefabs -> UI -> PanelLLM.prefab) as a child of the ProviderPanelParenet:
The game view now looks like this:
Update some text for clarity:
NLPBase is an abstract base class that acts as a common interface for all NLP systems and enforces the implementation of key methods required for the NLP system to function.
It contains the fields:
It contains the methods:
It contains the abstract methods:
These abstract methods are mandatory and are implemented by each derived system.
This abstract base class allows all NLP systems to behave in the same general way by having the same field and method names referring to context specific logic per derived NLP system (more on this with ExampleLLM.cs).
More on the key methods below.
SystemInit():
The SystemInit() method is an override of the RideSystemMonoBehavior method of the same name. This is inherited from the NLPBase class as it derives from RideSystemMonoBehavior. This override is optional, but is a good place to initialize the uri, authorization key and the initial prompt used for the provider if there is one.
AskQuestion(string question, Action<NLPResponse> onComplete):
The AskQuestion method is a required override of the NLPBase abstract method. This is a functionality implementation of the INLPQnASystem interface that NLPBase derives from. The purpose of this method is to store the user input into a context appropriate data structure and pass it to Request. It automatically stores the user input in the pastUserInput list for future access.
Request(string uri, string question, Action<NLPResponse> onComplete, string data = null)
The Request method is a required override of the NLPBase abstract method. This is a functionality implementation of the INLPSystem interface that NLPBase derives from. The purpose of this method is to request a response, wait for the response and deserialize that response from the NLP system into a value that can be easily referenced/used. It automatically stores the deserialized response in the generatedResponses list for future access.
Create a script for the new Anthropic system. For the purposes of this tutorial, it will be named AnthropicClaude.cs
Note: during the next few implementation steps, comparisons can be made to the GPT4 system.
Script Setup First Steps:
Addressing the Error:
There will be an error with the class. In particular the error stems from failing to implement the methods that are required when deriving from NLPBase: AskQuestion and Request. Luckily there is a simple way of doing this. Right click the class name -> Quick Actions and Refactorings… -> Implement abstract class. This will set up the required methods with the proper declarations automatically!
As with OpenAI GPT-4, Anthropic can receive a REST message and send a response in JSON format.
https://docs.anthropic.com/claude/reference/complete_post provides the required and optional headers and parameters.
In addition to the required authorization key in the header, the required parameters are:
Known JSON structures are typically (de)serialized with supporting classes. See \Assets\Ride (local)\Systems\NLP\OpenAIGPT3.cs for an example.
Key Elements of the AskQuestion Method:
Code:
public override void AskQuestion(string question, Action<NLPResponse> onComplete)
{
AddUserInput(question);
var conversationHistory = new List<string>();
for (int i = 0; i < pastUserInputs.Count; i++)
{
var userText = pastUserInputs[i];
string generatedResponse = "";if (i < generatedResponses.Count)
generatedResponse = generatedResponses[i];conversationHistory.Add($"\nHuman: {userText}\nAssistant: {generatedResponse}");
}string historyText = string.Join(" ", conversationHistory);
string data = JsonConvert.SerializeObject(new
{
model = "claude-1",
prompt = historyText,
max_tokens_to_sample = 256
});Request(m_uri, question, onComplete, data);
}
Key Elements of the Request Method:
Code:
public override async void Request(string uri, string content, Action<NLPResponse> onComplete, string data = null)
{
using var request = new UnityWebRequest(m_uri, "POST");
byte[] bodyRaw = Encoding.UTF8.GetBytes(data);
request.uploadHandler = new UploadHandlerRaw(bodyRaw);
request.downloadHandler = new DownloadHandlerBuffer();
request.SetRequestHeader("Content-Type", "application/json");
request.SetRequestHeader("x-api-key", m_authorizationKey);
request.SetRequestHeader("anthropic-version", "2023-06-01");var operation = request.SendWebRequest();
while (!operation.isDone)
{
await Task.Yield();
}if (!(request.result == UnityWebRequest.Result.ConnectionError))
{
var result = request.downloadHandler.text;
var res = JsonConvert.DeserializeObject<AnthropicResponse>(result);
AddGeneratedResponse(res.completion);
onComplete?.Invoke(new NLPResponse(res.completion));
}
}private class AnthropicResponse
{
public string completion { get; set; }
public string stop_reason { get; set; }
public string model { get; set; }
}
Note: the stopwatch of the NLPBase can be used in request to get response time and store it in m_responseTime to use later, but ExampleLLM.cs uses the stopwatch outside of the class explicitly instead.
Now that we have the functionality for asking for and receiving responses, we need to initialize the system to be able to make these requests.
Key Elements of the SystemInit Method:
Code:
public override void SystemInit()
{
var configSystem = api.systemAccessSystem.GetSystem<RideConfigSystem>();
m_uri = configSystem.config.anthropicClaude.endpoint;
m_authorizationKey = configSystem.config.anthropicClaude.endpointKey;AddUserInput(initialPrompt);
AddGeneratedResponse("Initial prompt registered successfully");
base.SystemInit();
}
Back in the ExampleLLM scene, we can now add our AnthropicClaude component to the previously created PanelLLM object and reference it in the provider field of the PanelLLM component.
Add the same prompt as used with GPT4:
You are a helpful assistant as part of the Virtual Human Toolkit 2.0 (VHToolkit 2.0). The VHToolkit 2.0 is developed by the USC Institute for Creative Technologies. It is currently in exclusive preview, only usable by a select lucky few. Any user should feel special about being able to use this preview before its formal release later this year. The VHToolkit 2.0 combines a flexible architecture with a principled API, using both academic and industry technologies. Capabilities include audio-visual sensing, speech recognition, natural language processing, nonverbal behavior generation and realization, and text-to-speech. Initially, many of these are implemented through the use of web services, with local alternatives being added in the future. The main game engine target is Unity, with Unreal Engine and Metahuman support underway. Your answers should be in a conversational tone with single sentence responses, emphasizing brevity while summarizing when necessary. Responses should be in spoken language, no characters that cannot be spoken.
You’re done! You should be able to interact with both GPT4 and Clause simulteanously now.
Adding the panel as a child of the ProviderPanelParent allows for the scene to prompt multiple NLP systems at once.
Fields of ExampleLLM:
Methods of ExampleLLM:
AskQuestion can function in this polymorphic manner because the NLP systems both derive from NLPBase which allows for the same method and field names to be used to reference the context specific implementation of any of the derived NLP systems within a single loop.
|
Achievement Unlocked! What other cloud solutions might you implement within VHToolkit 2.0 to benefit your research and simulations? |
