Giving Virtual Reality Characters a Voice

Problem

We worked with a fast growing med-tech company building a virtual reality platform to simulate doctor-patient interactions. Together, we developed a voice interface to allow users to have a naturalistic conversation with virtual reality patients.

Our initial implementation was based on the open source chatbot framework RASA. At the time, RASA implemented a traditional chatbot approach using models to identify the user’s intent, and any entities in the user input such as names, dates, locations, etc.

This approach worked well, but we found the RASA framework was unnecessarily complicated, and it quickly became a bottleneck when resolving issues in existing training scenarios, and the developing new ones.

Solution

To resolve this bottleneck, we abandoned RASA and developed custom “transformer” models for identifying intents and entities, using the same broad class of architecture that is behind the latest developments in large language models (LLMs) such as ChatGPT.

We deployed these models using a third party service called Inference Endpoints, developed by Hugging Face. Hugging Face inference endpoints make deployment easier and reduce maintenance costs.

To integrate the models into the client’s platform, we developed a custom API that connects to those models and applies some business rules before communicating back to the platform.

Another bottleneck we encountered was the speed with which training data could be generated or fixed by the client, so we implemented an LLM based data generation tool with a human in the loop to assess the quality.

Impact

The custom transformer models improved the performance of the solution compared to the RASA models. The move to Inference Endpoints and implementation of the LLM data generation solution reduced the time taken for the client to iterate and develop new flows for their users, and generate and curate data. It has enabled them to put changes into production in a matter of minutes.