3d hand modeling, animation in XNA games studio

Virtual 3D hand Modeling

To facilitate the communication between the Deaf and hearing, highly skilled interpreters have traditionally been used. However, these interpreters tend to be very costly and it is a great effort to become a good interpreter to translate between a spoken language and a sign language correctly and efficiently. The use of an interpreter is not always appropriate and they need to be notified in advance when their services are to be required [1]. Another important factor to consider is that there will simply never be enough good trained interpreters that can assist the millions of Deaf [1]. A Machine Translation (MT) system that can translate between a verbal language such as English and a Sign Language that employs a three dimensional (3D) computer generated virtual human will solve the above problem of insufficient interpreters. Such a machine translation system can be used in many different applications such as: Deaf telephony, English and Sign Language education and whenever an interpreter is required [2].

The human hands are complex articulated structures with multiple degrees of freedom. This makes the modeling and animation of high quality flexible virtual hands extremely difficult especially for real-time interactive applications. We wish to employ virtual hands for real-time Sign Language visualization for which they are of the utmost importance [3].

Bones and Joints of a Human Hand

Figure 5.1 Bones of the right hand (dorsal view) [4]

The skeleton of a hand can divided into three main sections namely the carpal bones (carpals), metacarpal bones (metacarpals) and phalangeal bones (phalanges), as can be seen in Figure 5.1. There is a total of 8 carpals which are joined together and form part of the wrist and base of the hand. The carpal have negligible rotational freedom and is connected to the 5 metacarpals of the thumb and 4 fingers (index, second, third and fourth) by carp metacarpal joints (CMC). The range of movement of the CMC joints cannot be easily measured and it is noted by [5] that only the CMC joints of the thumb, third and fourth finger metacarpals are rotational. The CMC joints of the third and fourth finger metacarpals have small rotational freedom and allows for a flexible palm. The CMC joint of the thumb metacarpal has 3 Degree of Freedom(DOF)s and allows for a dexterous thumb that enable us to grasp and hold objects properly. The metacarpals are in turn connected to the proximal phalanges of the thumb and 4 fingers by the metacarpophalangeal joints (MCP). The MCP joints of the 4 fingers, which are commonly known as the knuckle joints, have much greater rotational freedom than the CMC joints of the 4 fingers and are noted by [5] to have 3 DOFs each. The 4 proximal phalanges of the 4 fingers are connected to 4 middle phalanges by proximal interphalangeal joints (PIP). These 4 middle phalanges are in turn connected to distal phalanges by distal interphalangeal (DIP) joints. The proximal phalanx of the thumb is connected to a distal phalanx by an interphalangeal (IP) joint. All the PIP, DIP and IP joints have 1 DOF each. Should we take the third and fourth finger CMC joints to have 3 DOFs and the MCP joint of the thumb to have 2 DOFs, all of the above results in a total of 27 bones and a total of 32 DOFs [5]. This can make any animator think twice before modeling and animating realistic hands.

Table 5.1 DOF of the Joints in Degree (°) [9].

Modeling and Skinning

To model virtual humans or hands in our case, we decided to make use of 3D Studio Max (3dsMax) from Autodesk that is easy to use, innovative and professional software for the modeling of 3-Dimensional characters [7]. One of the goals of 3dsMax is to develop an anatomically correct model that has only the necessary number of vertexes and is optimized for animation. By parametersing a base polygonal mesh, 3dsMax enables a variety of virtual characters to be easily modeled by manipulating parameters through its graphical interface. We modeled a virtual human hand with different fingure segment lengths in 3dsMax.After modeling, the meshes was exported as 3ds object models and imported into FX Composer [8]. FX Composer, another famous software tool, is a 3D content creation suite for modeling, shading, rendering, compositing and surface editing applications [8].

3D Studio Max

Figure 5.2 3D Studio Max

Autodesk 3ds Max, formerly 3D Studio MAX, is a modeling, animation and rendering package developed by Autodesk Media and Entertainment. It has modeling capabilities, a flexible plugin architecture and is able to be used on the Microsoft Windows platform. It can be used by video game developers, TV commercial studios and architectural visualization studios. It is also used for movie effects and movie pre-visualization.

In addition to its modeling and animation tools, the latest version of 3ds Max also features advanced shaders (such as ambient occlusion and subsurface scattering), dynamic simulation, particle systems, radiosity, normal map creation and rendering, global illumination, an intuitive and fully-customizable user interface, and its own scripting language.

FX Composer

Figure 5.3 FX composer

FX Composer is a powerful integrated development environment for shader authoring. With support for DirectX and OpenGL, HLSL, COLLADA FX, and CgFX, as well as .fbx, .x, .3ds, .obj, and .dae formats, FX Composer provides true cross-platform functionality.

XNA games Studio

Figure 5.4 XNA Games Studio Logo

Microsoft XNA is a set of tools with a managed runtime environment provided by Microsoft that facilitates computer game development and management [6].

The XNA Framework is based on the native implementation of .NET Compact Framework 2.0 for Xbox 360 development and .NET Framework 2.0 on Windows. It includes an extensive set of class libraries, specific to game development, to promote maximum code reuse across target platforms. The framework runs on a version of the Common Language Runtime that is optimized for gaming to provide a managed execution environment.

Animating a 3D model on Visual Studio using XNA framework

Creating a hand object on 3d space

Defining 3D space and camera vector on defined Graphic device

Vector3 cameraPosition = new Vector3(0.0f, 60.0f, 160.0f);

Vector3 cameraLookAt = new Vector3(0.0f, 50.0f, 0.0f);

Matrix cameraProjectionMatrix;

Matrix cameraViewMatrix;

spriteBatch = new SpriteBatch(GraphicsDevice);

cameraViewMatrix = Matrix.CreateLookAt(

cameraPosition,

cameraLookAt,

Vector3.Up);

cameraProjectionMatrix = Matrix.CreatePerspectiveFieldOfView(

MathHelper.ToRadians(45.0f),

graphics.GraphicsDevice.Viewport.AspectRatio,

1.0f,

10000.0f);

Loading and Drawing Hand Object on 3D Space

GameObject hand = new GameObject();

hand.model = Content.LoadModel("Models\\Hand");

hand.scale = 5f;

hand.position = handBase.position + new Vector3(0.0f, 20.0f, 0.0f);

DrawGameObject(hand);

Figure 5.5 Modeled hand on Visual Studio 2008, XNA Framework

Defining DOF constrains in axis

hand.rotation.Y = MathHelper.Clamp(

hand.rotation.Y,

-MathHelper.PiOver4, MathHelper.PiOver4);

hand.rotation.X = MathHelper.Clamp(

hand.rotation.X,

0, MathHelper.PiOver4);

References

[1] DEAFSA. Deaf Federation of South Africa, http://www.deafsa.co.za.

[2] M. Huenerfauth, “Generating American Sign Language Classifier Predicates

For English-To-ASL Machine Translation,” Ph.D. dissertation,

Computer and Information Science, University of Pennsylvania, 2006.

[3] High Quality Flexible H-Anim Hands for Sign

Language Visualisation

Desmond E. van Wyk, James Connan

Department of Computer Science

University of the Western Cape, Private Bag X17 Bellville, 7535, South Africa

[4] The Visual Dictionary. Volume 3: Human body, BONES OF THE

HAND (dorsal view), http://www.infovisual.info/03/027 en.html.

[5] I. Albrecht, “Faces and Hands: Modeling and Animating Anatomical and Photorealistic Models with Regard to the Communicative Competence of Virtual Humans,” Ph.D. issertation, Ph. D. thesis, Universitat des Saarlandes (December, 2005).

[6] http://en.wikipedia.org/wiki/Microsoft_XNA

[7] http://en.wikipedia.org/wiki/3D_studio_max

[8] http://developer.nvidia.com/object/fx_composer_home.html

[9] High Quality Flexible H-Anim Hands for Sign Language Visualisation Desmond E. van Wyk, James Connan Department of Computer Science University of the Western Cape, Private Bag X17 Bellville, 7535, South Africa

Abbreviations

MT-Machine Translation

3D - Three Dimensional

CMC-Carp metacarpal joints

DOF-Degree of Freedom

MCP-Mmetacarpo Phalangeal joints

PIP-Proximal Inter Phalangeal joints

DIP-Distal Inter Phalangeal joints

3dsMax-3D Studio Max