Audio To Lip sync

With the MetaHumanSDK plugin creating lip-sync animations is now a breeze.

In-editor usage

To create an animation from an audio file using the editor, you need to do the following:

1. in the Content Browser select the sound asset you want to generate the animation from (maximum length of audio file 5 minutes);

2. from the context menu select Create Lipsync Animation tab.

After that the window pops up.

Select the Skeleton for which the animation is generated.

If you use the default settings, the animation will be generated for ArKit-FACS blendshapes.

Apple Developer Documentation

To make the generated animations work with Metahuman, you need to apply mapping first. Luckily the plugin does this for you. in order to generate a Metahuman-compatible lipsync animation choose EMetahuman in the MappingMode tab.

By clicking Generate you get an animation compatible with Metahuman.

In addition, you can start using the animation in the sequencer. To get the best result make sure that the start of the soundtrack and the animation are time-synchronized.

Moreover, you can start using animation directly in your scene.

Runtime generation

As an option you can generate lip sync animation while the application is running.

Let's have a look at the metahuman mapping config. In order to make your animation work on metahuman, apply the following mapping configurations.

Pose Asset	mh_dhs_mapping_anim_PoseAsset
Mapping Asset	None
Bone Mapping Asset	None
Setup for Metahumans	Enable

Short delay runtime generation

When using lip sync generation in runtime, you may encounter the problem with response time. The thing is that its duration directly depends on the length of the incoming audio file.

To reduce the playback delay, we suggest use short-delay implementation of audio-to-lipsync method. This method allows you to get the animation in chunks as they are generated, without waiting until the file is fully processed.

During the request execution a buffer will be created where the animation chunks are added as the data is received. When the buffer is full enough to start playing, the OnFilled event is triggered. After that you can start playing the resulting animation sequentially, selecting the necessary chunks from the buffer.