|

|  How to Integrate Microsoft Azure Cognitive Services with Unreal Engine

How to Integrate Microsoft Azure Cognitive Services with Unreal Engine

January 24, 2025

Learn to connect Microsoft Azure Cognitive Services with Unreal Engine for enhanced AI capabilities and interactive gaming experiences.

How to Connect Microsoft Azure Cognitive Services to Unreal Engine: a Simple Guide

 

Set Up Microsoft Azure Account

 

  • Navigate to the Azure Portal and create an account if you haven’t already.
  •  

  • Create a new resource and select "Cognitive Services". You can choose the specific type of service you need, such as Text Analytics, Speech to Text, etc.
  •  

  • Once the resource is created, note down the subscription key and endpoint URL.

 

 

Prepare Unreal Engine Project

 

  • Open Unreal Engine and start a new or existing project where you want to integrate Azure Cognitive Services.
  •  

  • Ensure your project supports HTTP requests by enabling the HTTP module. Add the following to your project's `.Build.cs` file:

 

PublicDependencyModuleNames.AddRange(new string[] { "HTTP", "Json", "JsonUtilities" });

 

  • Compile your Unreal Engine project to ensure all modules are included.

 

 

Integrate Microsoft Azure Cognitive Services

 

  • Create a new C++ class in Unreal Engine, perhaps called `AzureCognitiveServiceClient`, to handle the integration.
  •  

  • Within this class, set up HTTP requests to interact with Azure Cognitive Services. This will typically involve using Unreal’s `FHttpModule` to create a POST request.
  •  

 

#include "HttpModule.h"
#include "IHttpRequest.h"
#include "IHttpResponse.h"
#include "JsonUtilities.h"

void UAzureCognitiveServiceClient::SendRequest(const FString& RequestData)
{
    TSharedRef<IHttpRequest> Request = FHttpModule::Get().CreateRequest();
    Request->OnProcessRequestComplete().BindUObject(this, &UAzureCognitiveServiceClient::OnResponseReceived);
    Request->SetURL("<Your Endpoint URL>");
    Request->SetVerb("POST");
    Request->SetHeader("Content-Type", "application/json");
    Request->SetHeader("Ocp-Apim-Subscription-Key", "<Your Subscription Key>");
    Request->SetContentAsString(RequestData);
    Request->ProcessRequest();
}

void UAzureCognitiveServiceClient::OnResponseReceived(FHttpRequestPtr Request, FHttpResponsePtr Response, bool bWasSuccessful)
{
    if (bWasSuccessful && Response->GetResponseCode() == 200)
    {
        FString ResponseBody = Response->GetContentAsString();
        // Parse JSON response
    }
    else
    {
        // Handle error
    }
}

 

  • Use JSON utilities to parse the response from Azure. The response content must be converted from JSON format to the type you need.

 

 

Testing and Debugging

 

  • Ensure that your Unreal Engine project can access external networks since the HTTP requests will connect to Azure Services.
  •  

  • Test the integration by running the project and invoking the Azure service functions. Use UE\_LOG for debugging to display meaningful logs for request successes or failures.
  •  

  • Check Azure’s portal to monitor the usage and verify if the requests from Unreal Engine are being received correctly.

 

Omi Necklace

The #1 Open Source AI necklace: Experiment with how you capture and manage conversations.

Build and test with your own Omi.

How to Use Microsoft Azure Cognitive Services with Unreal Engine: Usecases

 

Immersive Real-Time Language Translation

 

  • Leverage the power of Microsoft Azure Cognitive Services' Language APIs for real-time language translation and speech recognition.
  •  

  • Integrate with Unreal Engine to create a highly interactive and immersive environment where users can communicate naturally in different languages.
  •  

  • Make immersive virtual collaboration platforms for diverse international teams, ensuring seamless communication irrespective of linguistic barriers.

 

Setup and Integration

 

  • Set up Azure Cognitive Services and acquire the necessary API keys for language and speech services.
  •  

  • Integrate API calls in Unreal Engine using Blueprints or C++ to handle language translation and speech recognition.
  •  

  • Utilize Unreal Engine's robust rendering and audio management capabilities to deliver seamless translated audio and text feedback to users.

 

Enhanced User Interaction

 

  • Create realistic virtual environments where users from different linguistic backgrounds can interact using their native languages.
  •  

  • Enable voice commands and responses in different languages, enhancing the accessibility and usability of Unreal Engine applications.
  •  

  • Implement subtitled dialogues or translated voiceovers automatically using the translated data from Microsoft Azure.

 

Testing and Deployment

 

  • Conduct rigorous testing to ensure that language translations and speech recognition are accurate and responsive within the Unreal Engine environment.
  •  

  • Deploy the final product to multiple platforms, leveraging Azure's scalability to handle potentially high loads of translation requests in real time.
  •  

  • Collect user feedback and use Azure's AI capabilities to continually improve the language model and enhance user experiences.

 


// Unreal Engine C++ sample code snippet to call Azure translation service

FString Endpoint = "https://api.cognitive.microsofttranslator.com/translate?api-version=3.0&to=es";
FString ApiKey = "YOUR_AZURE_API_KEY";

TArray<TSharedPtr<FJsonValue>> JsonArray;
TSharedPtr<FJsonObject> JsonObject = MakeShareable(new FJsonObject);
JsonObject->SetStringField(TEXT("Text"), TEXT("Hello, how are you?"));
JsonArray.Add(MakeShareable(new FJsonValueObject(JsonObject)));

FString JsonPayload;
auto Writer = TJsonWriterFactory<>::Create(&JsonPayload);
FJsonSerializer::Serialize(JsonArray, Writer);

FHttpModule* Http = &FHttpModule::Get();
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> Request = Http->CreateRequest();
Request->SetURL(Endpoint);
Request->SetVerb("POST");
Request->SetHeader(TEXT("Content-Type"), TEXT("application/json"));
Request->SetHeader(TEXT("Ocp-Apim-Subscription-Key"), ApiKey);
Request->SetContentAsString(JsonPayload);
Request->ProcessRequest();

 

 

Dynamic Character Emotion Recognition

 

  • Utilize Microsoft Azure Cognitive Services' Emotion APIs to analyze human emotions in real-time through facial expressions and voice tone.
  •  

  • Integrate these emotional insights into Unreal Engine to create dynamic and emotionally responsive virtual characters in games or simulations.
  •  

  • Enhance storytelling in virtual environments by allowing characters to react dynamically to user emotions, creating a deeply engaging narrative experience.

 

Setup and Integration

 

  • Configure Azure Cognitive Services and obtain API keys for emotion detection and analysis services.
  •  

  • Employ the Azure SDK to fetch real-time emotional data, integrating these data points into Unreal Engine using Blueprints or C++.
  •  

  • Leverage Unreal Engine's animation and behavior system to adjust character expressions and movements based on detected emotional cues.

 

Enhanced Player Experience

 

  • Develop immersive storylines where game characters respond to players' emotional states, offering personalized experiences.
  •  

  • Enable interactive educational simulations where virtual instructors adjust their teaching style based on student reactions and emotions.
  •  

  • Facilitate therapeutic virtual environments with emotive content responding adaptively to user well-being states.

 

Testing and Iteration

 

  • Perform extensive testing to verify the accuracy and timeliness of emotion recognition within the Unreal Engine framework.
  •  

  • Iterate on emotional response algorithms to fine-tune character reactions for enriched player engagement and realism.
  •  

  • Implement continuous feedback loops using Azure's machine learning to enhance character response accuracy over time.

 


// Unreal Engine C++ sample code snippet to integrate Azure emotion recognition

FString Endpoint = "https://<region>.api.cognitive.microsoft.com/emotion/v1.0/recognize";
FString ApiKey = "YOUR_AZURE_API_KEY";

TSharedPtr<FJsonObject> JsonObject = MakeShareable(new FJsonObject);
JsonObject->SetStringField(TEXT("Url"), TEXT("https://example.com/image.jpg"));

FString JsonPayload;
auto Writer = TJsonWriterFactory<>::Create(&JsonPayload);
FJsonSerializer::Serialize(JsonObject.ToSharedRef(), Writer);

FHttpModule* Http = &FHttpModule::Get();
TSharedRef<IHttpRequest, ESPMode::ThreadSafe> Request = Http->CreateRequest();
Request->SetURL(Endpoint);
Request->SetVerb("POST");
Request->SetHeader(TEXT("Content-Type"), TEXT("application/json"));
Request->SetHeader(TEXT("Ocp-Apim-Subscription-Key"), ApiKey);
Request->SetContentAsString(JsonPayload);
Request->ProcessRequest();

 

Omi App

Fully Open-Source AI wearable app: build and use reminders, meeting summaries, task suggestions and more. All in one simple app.

Github →

OMI NECKLACE + OMI APP
First & only open-source AI wearable platform

a person looks into the phone with an app for AI Necklace, looking at notes Friend AI Wearable recorded a person looks into the phone with an app for AI Necklace, looking at notes Friend AI Wearable recorded
a person looks into the phone with an app for AI Necklace, looking at notes Friend AI Wearable recorded a person looks into the phone with an app for AI Necklace, looking at notes Friend AI Wearable recorded
online meeting with AI Wearable, showcasing how it works and helps online meeting with AI Wearable, showcasing how it works and helps
online meeting with AI Wearable, showcasing how it works and helps online meeting with AI Wearable, showcasing how it works and helps
App for Friend AI Necklace, showing notes and topics AI Necklace recorded App for Friend AI Necklace, showing notes and topics AI Necklace recorded
App for Friend AI Necklace, showing notes and topics AI Necklace recorded App for Friend AI Necklace, showing notes and topics AI Necklace recorded