The SILVIA Gesture Controller Component

The Gesture Controller component controls the use of the various IK control subsystems. The values in it generally are not set directly, but most are public fields, so they can be inspected in Editor, and at runtime. Generally, Gesture Controller values are set from the Pose Sequencer, together they form the core of the Natural Motion Synthesis system. Using an analogy to a Moog Synthesizer for example, the Pose Sequencer would be all the dials, patch bays, switches, and faders, and the Gesture Controller would be the keyboard.

Gesture Controller uses the built in IK system of the Mecanim Animator and the classic Rigidbody and Capsule Collider setup to interact with Unity’s physics such as with gravity to keep the Avatar from falling through the surface below. The useGrounded bool turns the physics gravity off and on.

Under the header Switch Subsystem Off and On are several bool checkboxes. The useIKGestureRig bool turns off and on the ArmsControlRig. This rig controls the motion of the arms via the HandIKGoals and uses a setup to provide a more natural curved motion to the IK Goals transitions from position to position. If you wish to control the arms via Mecanim or move the IK Goals around via code then turn this bool false by unchecking the box.

This could be useful in speech output where a particular complex gesture of the arms emphasizes the content of that phrase. Once the phrase has finished the complex gesture not being appropriate to the next phrase the control can be returned to the ArmsControlRig, setting the bool to false.

The positioning of the Hand IK Goals overrides the positioning set by the ArmsControlRig when useMecanimMasterIK is set to true. When set to false the Hand IK Goals positioning control reverts back to the ArmControlRig Avatar subsystem.

The alignPelvisWithPelvicTarget uses the built in Mecanim IK to set the BodyPosition to the PelvicGoal which controls the linear translation of the rig Pelvis left-right, up-down and back-forth. The PelvicTarget is a child of the PelvicGoal and it controls the Mecanim IK BodyRotation.

The useAutoElbowHints turns on and off the calculation to procedurally place the elbow hint using the shoulder-wrist-elbow triangle plane. Setting the useAutoElbowHints to false gives control of the position to the leftElbowHintElevation, leftElbowHintMidline, rightElbowHintElevation and rightElbowHintMidline values which can be inspected in the Transforms For Arms Rig section of this component.

The values for the Arms Gesture Rig can be inspected and references to the MecanimMasterGestureRig puppeteer and its IK Goals can be clicked on to take you to them in the Hierarchy view.

The next section of the Gesture Controller component addresses the Pelvis as well as the Spine Bone Chain positions and rotations. The Spine Bone Chain distributes the spineTilt, spineBend and spineTwist rotations along the bone chain.

The Neck and Head poses comprise the next section of this component. Though Neck and Head are a part of the BodyExpression subsystem the Neck and Head pose as well as the body orientation can be controlled by the LookAtObject position using Mecanim IK and setting the weights. The LookAtWeight controls how much the eyes move from default or current pose to stare at the LookAtObject. The HeadLookWeight controls how much the head participates in facing the LookAtObject. The BodyLookWeight value controls how much the upper body participates in facing the LookAtObject.

The Look Weight values can be set directly from code or you can set it in a PAGER tag in a SILVIA response as per the following example which sets the headLookWeight to 0.75.

 <GestureCtrl-FLOAT headLookWeight,params=?0.75/>

The next section of the component addresses the IK Goals and weighting for hands and feet. References to the IK Goal and Hint transforms can be clicked on to take you directly to them in the Hierarchy. The position and rotation strengths per goal and hint can be inspected and set here. The Avatar IK Rig is set up with Wrist Rotator helper bones that rotate the mesh or the lower half of the forearm when the wrist is rotated. This allows more precision control over the Hand Pose, doesn’t cause nasty mesh break artifacts at the wrist joint and is much more predictable and controllable than the built in Hand IK rotation. The HandIKRotationStrength values are set to 0 so they have no effect on the hand rotations via IK but uses the positioning strength at 1.0 to follow to IK Goal positions exactly or as close as the limb extension and constraints allow. By controlling the position strength the hands will try to reach the IK goals from their default positions. This can be used for example in setting up the IK Goals for a wide expressive gesture and lerping the strengths from 0 to 1.0 which would move the hands from their default positions towards the wide expression gesture positions.

In the next section are the references to the Hand Rig Control and Arm Rig Control transforms. The Left Hand Rig controls positioning of the Left Hand IK Goal. The Left Elbow is used to align the rig with the forearm along the elbow to wrist vector. The Right Forearm Twist and Left Forearm Twist bones are referenced here though their calculations are carried out in the SilviaHandPoseController component and controlled by the leftWristTwist and rightWristTwist values.

The Left Clavicle and Right Clavicle bones and their default localRotation are recorded at Start so any rotations added for shoulder shrugs, arm raising above shoulder line or reaching forward and down can be zero-ed out and applied rotations then make sense in human readable terms.

The LeftArmControlRig transform is a child of the ProceduralGestureControlRig parent which is a child of the PelvicTarget. This gives the control rig the same up and forward vectors as the rig at the base of the spine. The root of the LeftArmControlRig follows the shoulder pivot position while remaining aligned with the pelvis. The swing from front to rear and raising and lowering of the arm are under control of the LeftArmRotator which is a child of the LeftArmRig. The extension away from and towards the body are controlled by the LeftArmExtender which is a child of the LeftArmRotator. The LeftHandRig position is synched to the LeftArmExtender. The same hierarchical setup is used for the RightArmRig.

This system allows for more natural non-linear motion when interpolating arms gesture values. It also brings secondary motion to the arms and hands. For example and assuming the arm rigs are not interpolating control values but have static values and you have the Avatar bending forward via rotation of the PelvicTarget. You would also have the spine chain bending over as well for a more natural pose. Since the ArmControlRigs are attached to the shoulder but maintain the PelvicTarget orientation the difference in rotations up the spine bone chain accumulate. The consequence is that the position of the Extender, though static within the ArmControRig, moves the hand position in relation to the shoulder. The arm is not unnaturally locked to a rotation nor moves linearly as it would if following an interpolated IK Goal.

The ElbowHint midline and elevation values control the position of the left and right ElbowHint objects in relation to the pelvis when the useAutoElbowHints is set to false.

The nextsectionofthecomponentisHeadControl.Thoughtheheadandneckrotationscanbecontrolled from BodyExpression values the weighting of the headLook and bodyLook strengths as multiplied by the lookAtStrength will override head, neck and body rotation values. The default lookAtObj is a child of the AvatarCamera and its initial position is recorded and all offsets are in addition to the initial offset. If the camera is moved the Avatar will still look towards it as is general with most speaking or narrating performances. If the performance requires the Avatar to look or glance at a particular object or point in space to enhance a phrase.

Transforms for the leg rig and foot rig controls as well as their current values are accessed and displayed for inspection in this section. The leg rig consists of a root that is attached at the thigh/pelvis joint pivot with its local up and forward matching the Avatar pelvis. As a child of this is a rotator object that swings or rotates from front to rear as well as rotating the leg towards and away from the body midline. As a child of this is an extender object that rides the rotator Z axis to contract the leg towards or extend it away from the pelvis. The foot rig is a separate setup that attaches its position to the leg rig Extender transform with its up vector defaulting to align with the extender transforms Z axis. All subsequent foot and ankle rotations are multiplied by this quaternion. This gives the foot stability during natural motion and allows control of the foot rig outside of the Avatar procedural idle and pose to pose sequencing. This subsystem is handled slightly differently as generally during a speaking performance or interactive dialogue the Avatar will not be continually moving feet around as they would with arms, hands, head etc.

The Body Extension section calculates the total height of the Avatar as well as the length of the limb bone chains. This will be important data as the system is expanded to load several variant Avatars and reconstruct the natural motion rigs.