Facial Musculoskeletal System

[Rex Brooks <rexb@starbourne.com>]

Title: Facial Musculoskeletal System

Greetings H-Anim Folks,

I haven't checked with you in a long time, but I have finally finished building the first version of a face skeleton system using 3DSMax, and I wanted to see if there are any of you who might be interested in pursuing this further.

If you go to http://www.starbourne.com I have set my website up so that you can easily go directly to the examples and explanatory material on this work. I have overlapped a ppt presentation with images and avi animations that show the system at work.

What I have created is a system which would eliminate the need to build individual morph-targets to animate indexed point sets for facial animation for every individual avatar. It also can automate facial expressions using Human Markup Language for emotion-states that correspond to expression-states using the face skeleton elements. It uses HumanMarkup which is XML so XML streaming can be used for interactivity. This is specifically aimed at avoiding any dependencies on MPEG-4 whether it is truly Royalty Free now or ever.

I am including the first draft of a proposal statement.

What I am  looking for is twofold-- to build a plug-in for Character Studio using the system I have built and outputing H-Anim 2001 avatars and developing browser plug-in player support for making this system an optional extension of H-Anim 2001 within X3D.

The next set of questions is if would be feasible then to move on to creating the scripts through interactive online interfaces to drive the expression behaviors using an adaptation of bvh files for facial reference points. Using motion capture for a kinesic library or vocabulary face and body animation that can be standardized for cultures, emotions, dance, etc is the purpose of this. Again, I intend to develop a system that is open source, public, royalty free and totally unencumbered.

The reason why I am specifying bvh files is that one of the members of our HumanMarkup TC at OASIS is doing motion capture that will be capable of building this kinesic vocabulary of movements that will, in turn, allow on-demand physical behaviors as outlined above. So it stands to reason that if we can script standard facial expressions, we can script overall avatar behavior using standard emotions and movements with a set of dashboard-menu iconic controllers for users to drive their own avatars.

Here is the first draft of the proposal statement. Please take a look at this stuff and let me know what you think. Obviously, if we could work together on a plug-in, I think it could lead to great things. I have made the additions to the H-Anim hierarchy in blue below.

Please excuse my difficulties in clearly expressing the components, and use the visual materials on my website to explore further what I am describing.

Ciao,

Rex

---

Copyright © 2003 Rex Brooks/Starbourne Communications Design

Facial Musculoskeletal Extension to H-Anim 2001 draft specification

I have developed this musculoskeletal system based on CharacterStudio in 3D Studio Max, using the Biped and Physique structural system which corresponds to a fairly standard bones system of joints and segments in a hirarchical arrangement of parent and child joint-segment relationships capable of advanced forward and inverse kinematics using envelopes of influence or weighting for polygonal mesh vertices controlled by joints and bone segments. So it corresponds to the H-Anim 2001 draft specification. I based the naming system of the face skeleton on the major facial muscle groups responsible for producing the six basic human emotional facial expressions of joy, sadness, fear, anger, disgust and surprise which seem to be fairly well accepted for the purposes of representing human facial expressions.

However, I took liberty in the area of producing eyebrow deformations of the facial mesh to allow for the more pictographic purposes of changing the angles of the visible brow line shapes upward or downward through the midpoint over the eye orbit from the outer and inner points and the drawing-together or bunching of the brows over the nose.

These results don't easily result from a simple movement of a muscle analog segment in relation to an arbitrary anchor attachment to a bony analog geometric structure of the skull analog geometry. It was also not easy to simulate a muscle contraction which would most closely resemble an axial scaling of a segment provided the anchor attachment and skin surface mesh attachment, so I didn't follow the musculoskeletal analogy in that..

I discovered that I got better, more repeatable facial deformation results by refraining from establishing arbitrary anchors in multiple skull site attachments for the muscle analogs, which I modelled in two or more hierarchical components--usually as three-sided hollow cylinders for endsegments roughly corresponding to muscle portions attached to bony skull sites complemented with three-sided inner cylinders acting as pistons, but leaving both components movable while maintaining a parent-child relationship between outer cylinder and inner piston with outer segments parented by the skull or jaw. The Frontalis and Mentalis components worked better as cubes and a cubical cylinder-piston set respectively, while the orbicularis muscles a modeled witth three-sided hemitoroids. I also did not establish constraints for the any component's movement, scaling or rotation other than the link hierarchy which mostly employs an implied, but not specified sliding joint relationship. Rotational movements, while possible, did not deliver satisfactory results, nor did scaling in the very limited movements of facial vertices needed.

I did build a more complex set of links for the the eyebrow hierarchies which include specific pivot points in the CorrugatorNose to CorrugatorBrow to BrowInner segments for the inner portion of the chain and for the terminating BrowOuterEnd segments, all of which require segment-localized axial coordinate systems for individual segments with respect to parent-child relationships in the kinematic hierarchy chain. These are shown in the detail illustrations included with this proposal.

Naming Convention Note: I have used Inner and Outer as terms rather than using Proximal for the Inner "piston" segments to avoid confusion since this does not precisely translate to extant terminology, but I would have no objections to changing that so the outer-parent segment which uses no qualifier for the muscle analog pairs would then be the Distal component, but the eyebrow chain would still need more qualifying terms to be consistent and I think that the necessity for naming the left endsegment as LBrowDistalTerminator is a bit much compared to LBrowOuterEnd. However, since a tool would be constructed to make all those conversions for plug-ins to output H-Anim for the major modeling/animation packages it makes almost no difference, so the differences between my preferences for initial caps for stringing together compound terms made up of individual words without "-" or "_" also makes no difference. Cum si, Cum ça.

HumanoidRoot : sacrum
  sacroiliac : pelvis
  |   l_hip : l_thigh
  |     l_knee : l_calf
  |       l_ankle : l_hindfoot
  |         l_subtalar : l_midproximal
  |           l_midtarsal : l_middistal
  |             l_metatarsal : l_forefoot
  |   r_hip : r_thigh
  |     r_knee : r_calf
  |       r_ankle : r_hindfoot
  |         r_subtalar :  r_midproximal
  |           r_midtarsal : r_middistal
  |             r_metatarsal : r_forefoot
  vl5  : l5
    vl4 : l4
      vl3  : l3
        vl2 : l2
          vl1  : l1
            vt12 : t12
              vt11 : t11
                vt10  : t10
                  vt9 : t9
                    vt8 : t8
                      vt7 : t7
                        vt6  : t6
                          vt5 : t5
                            vt4 : t4
                              vt3 : t3
                                vt2 : t2
                                  vt1  : t1
                                    vc7 : c7
                                    | vc6 : c6
                                    |   vc5 : c5
                                    |     vc4  : c4

                                    |       vc3 : c3
                                    |         vc2  : c2
                                    |           vc1 : c1
                                    |             skullbase : skull
                                    |               l_eyelid_joint : l_eyelid(LEyelid)
                                    |               r_eyelid_joint : r_eyelid(REyelid)
                                    |               l_eyeball_joint : l_eyeball
                                    |               r_eyeball_joint : r_eyeball
                                    |               l_eyebrow_joint : l_eyebrow--CorrugatorNose
                                                                |   CorrugatorBrow-L
                                                                    |   CorrugatorBrow_joint:LBrowInner
                                                                     |       LBrowInner_joint:LBrow
                                                                  |               LBrow_joint:LBrowOuter
                                                                  |                       LBrowOuter_joint:LBrowOuterEnd
                                    |               r_eyebrow_joint : r_eyebrow--CorrugatorNose
                                                                   |   CorrugatorBrow-R
                                                                    |   CorrugatorBrow_joint:RBrowInner
                                                                     |       RBrowInner_joint:RBrow
                                                                  |               RBrow_joint:RBrowOuter
                                                                  |                       RBrowOuter_joint:RBrowOuterEnd
                                    |               globalskullattachment_joint : Frontalis-L
                                    |               globalskullattachment_joint : Frontalis-R
                                    |               globalskullattachment_joint : LevatorLabiiSuperioris-L
                                    |               globalskullattachment_joint: LevatorLabiiSuperiorisInner-L
                                    |               globalskullattachment_joint : LevatorLabiiSuperioris-R

                                    |               globalskullattachment_joint : LevatorLabiiSuperiorisInner-R
                                    |               globalskullattachment_joint : Zygomatic-L
                                    |               globalskullattachment_joint : ZygomaticInner-L
                                    |               globalskullattachment_joint : Zygomatic-R
                                    |               globalskullattachment_joint : ZygomaticInner-R
                                    |               globalskullattachment_joint : Risorius-L
                                    |               globalskullattachment_joint : RisoriusInner-L
                                    |               globalskullattachment_joint : Risorius-R
                                    |               globalskullattachment_jointt : RisoriusInner-L
                                    |               temporomandibular_joint : jaw
                                                       |       jaw_joint : Triangularis-L
                                              |       jaw_joint : TriangularisInner-L
                                        |       jaw_joint : Triangularis-R
                                              |       jaw_joint : TriangularisInner-R
                                        |       jaw_joint : Mentalis
                                            |       jaw_joint : MentalisInner
                                    l_sternoclavicular : l_clavicle
                                    | l_acromioclavicular : l_scapula
                                    |   l_shoulder : l_upperarm
                                    |     l_elbow : l_forearm
                                    |       l_wrist : l_hand
                                    |         l_thumb1 : l_thumb_metacarpal
                                    |           l_thumb2 : l_thumb_proximal
                                    |               l_thumb3 : l_thumb_distal
                                    |         l_index0 : l_index_metacarpal
                                    |           l_index1 : l_index_proximal
                                    |             l_index2 : l_index_middle
                                    |                 l_index3 : l_index_distal
                                    |         l_middle0 : l_middle_metacarpal
                                    |           l_middle1 : l_middle_proximal
                                    |             l_middle2 : l_middle_middle
                                    |               l_middle3 : l_middle_distal
                                    |         l_ring0 : l_ring_metacarpal

                                    |           l_ring1 : l_ring_proximal
                                    |             l_ring2 l_ring_middle
                                    |               l_ring3 : l_ring_distal
                                    |         l_pinky0 : l_pinky_metacarpal
                                    |           l_pinky1 : l_pinky_proximal
                                    |             l_pinky2 : l_pinky_middle
                                    |               l_pinky3 : l_pinky_distal
                                    r_sternoclavicular : r_clavicle
                                      r_acromioclavicular : r_scapula
                                        r_shoulder : r_upperarm
                                          r_elbow : r_forearm
                                            r_wrist : r_hand
                                              r_thumb1 : r_thumb_metacarpal
                                                r_thumb2 : r_thumb_proximal
                                                  r_thumb3 : r_thumb_distal
                                              r_index0 : r_index_metacarpal
                                                r_index1 : r_index_proximal
                                                  r_index2 : r_index_middle
                                                    r_index3 : r_index_distal
                                              r_middle0 : r_middle_metacarpal
                                                r_middle1 : r_middle_proximal
                                                  r_middle2 : r_middle_middle
                                                    r_middle3 : r_middle_distal

                                              r_ring0 : r_ring_metacarpal
                                                r_ring1 : r_ring_proximal
                                                  r_ring2 : r_ring_middle
                                                    r_ring3 : r_ring_distal
                                              r_pinky0 : r_pinky_metacarpal
                                                r_pinky1 : r_pinky_proximal
                                                  r_pinky2 : r_pinky_middle
                                                    r_pinky3 : r_pinky_distal

--

Rex Brooks
GeoAddress: 1361-A Addison, Berkeley, CA, 94702 USA, Earth
W3Address: http://www.starbourne.com
Email: rexb@starbourne.com

Tel: 510-849-2309

Fax: By Request