The Motion Synthesis System

For an NLP-based interactive Avatar, mimicking real human performances includes many aspects, such as postures, fidgeting motions with hands, facial ticks or expressions etc. As with humans, these would be somewhat random in appearance – in that you cannot precisely predict the next motion – but yet they are characteristic and within given ranges can portray motion and personality in a performance context.

The Motion Synthesis system works by basically selecting a range of motion and a time range to cycle through those motions per subsystem. These subsystems include Face Expressions, Hand Poses, Body Expressions, and IK, and are controlled by a procedural Inverse Kinematics system under the hood. Motion and Timing parameters are selected randomly between the ranges at each cycle per subsystem. This allows complex, but characteristic non-repetitive motion to be synthesized. This system allows for alive rigidity, lurching drunkenness – and anything between or beyond those – to be easily added to the Avatar’s performance. No keyframing of an animation timeline is needed. Performances are edited alongside the Exuder Text that they are synced to, assisting in rapid refinement of the performance portrayal.

This shows the system for setting up Time Ranges and Angle Limits per Body Expression subsystem.

To begin with, though a speaking performance is more focused on the head and face with the body posture and arm gestures conveying some part of that signal to the observer, the head postures just after delivering the speech add to the signal delivering further context. Think of Rodney Dangerfield and how just after the punchline to his latest wife joke would pop his eyes wide and snap his head randomly looking from here to there. His shuffling from foot to foot and nervous arm gestures with the tie yank added to the comedy performance to be sure, often evoking the term “Body Comic”. Control of the head and body is provided for in the Body Expression section but the LookAt section can completely override the torso and head using LookAt and setting their IK weight strength to 1.

In the Pose Sequencer component, you can set the useRandomLookAt to true by checking the box, which will select a random duration within the range of lookAtLerpDurationMinMax. When selecting the lookAtObj.position for the next interpolation it selects a random position within the ranges specified by the lookAtMinMaxX, lookAtMinMaxY, and lookAtMinMaxZ values. This allows a great variety of options to bring life to the Avatar performance. Values could be set minimal on the Y axis and large on the X range min-max values across the X axis to simulate randomly scanning the horizon. Large X and Y ranges can be given short duration ranges and look like Rodney Dangerfield scanning the audience for reactions after a punchline. The currLookAtObjPos and wantedLookAtObjPos allow inspection of the current from and to lerp (“Linear Interpolation”) values for the LookAt subsystem. “Lerping” allows smooth motion between positions.

The isSpeaking bool can be accessed for programmatic purposes. If isSpeaking is true then that means that the processing of the speech audio indicates non-silence. When it becomes false a speakTimer begins and when speakTimer exceeds the postSpeakWait duration the sentenceBreak bool becomes true. This is useful for triggering actions procedurally such as readjusting the cant of the head or body posture as often happens when a person finishes speaking.

Blinking is another subtle clue to signal an emotional state or context. Setting the useBlinking bool to true by checking the box sets a random blink interval to blink at. Control over the blinkDuration or how long the lids come together and open again, as well as the blink bool and Method can be had in the Face Rig Controller which we will discuss further on.

The Arms Gesture Idling section sets the ranges for gesture motions chosen randomly within those ranges. Gesture expression can be confined or expansive and targeted to either arm. Timings can be set up to be rapid or slow by adjusting the armsGestureIntervalMinMaxRange. When the timers per gesture motion are equaled or exceeded the timer gets reset and a new interval chosen in the range to drive the lerp from current to wanted values. The values below for example keep the Avatar arms extended and down by the side close to the body in a relaxed but attentive manner.

The following is an example of a PAGER tag that sets the range for the left arm up and down range during the performance.

<PoseSequencer-MINMAX leftArmGestUpDownIdleMinMax,params=?28,23/>

Within the tag < and /> the prefix prior to the dash – is the name of the subsystem to be addressed, in this example the PoseSequencer subsystem, specifically the MINMAX angle limits for the Left Arm.

The head and torso postures and motion can certainly add context to a speaking performance’s expressiveness and context as many signals are picked up by humans via subtle or exaggerated body motions and head movements. In the next section the min-max ranges for translation as well as rotational control of the pelvis along and around the major axes. During Procedural Idle mode each time one of the timers meets or exceeds its matched index interval it will pick a new value to lerp to in the min-max range of it’s also matched index value in a random time interval within the min-max range of the bodyExpIntervalMinMax.

The following is an example of a PAGER tag that sets the range for the rotation limits around the local up vector of the pelvis during the performance.

<PoseSequencer-MINMAX pelvicIdleTwistLimits,params=?-2,2/>