Machine Translation System

Java 3D Avatar Development for Wireless Sign Language Interpreter

10. Virtual 3D hand Modeling

10.1. Machine Translation System

To facilitate the communication between the Deaf and hearing impaired, highly skilled interpreters have traditionally been used. However, these interpreters tend to be very costly and it is a great effort to become a good interpreter to translate between a spoken language and a sign language correctly and efficiently. The use of an interpreter is not always appropriate and they need to be notified in advance when their services are to be required [1]. Another important factor to consider is that there will simply never be enough good trained interpreters that can assist the millions of Deaf [1]. A Machine Translation (MT) system that can translate between a verbal language such as English and a Sign Language that employs a three dimensional (3D) computer generated virtual human will solve the above problem of insufficient interpreters. Such a machine translation system can be used in many different applications such as: Deaf telephony, English and Sign Language education and whenever an interpreter is required [2].

The human hands are complex articulated structures with multiple degrees of freedom. This makes the modeling and animation of high quality flexible virtual hands extremely difficult especially for real-time interactive applications. We wish to employ virtual hands for real-time Sign Language visualization for which they are of the utmost importance [3].

10.1 Bones and Joints of a Human Hand

Figure 10.1 Bones of the right hand (dorsal view) [4]

The skeleton of a hand shown in Figure 10.1 can divided into three main sections namely the carpal bones (carpals), metacarpal bones (metacarpals) and phalangeal bones (phalanges), as can be seen in Fig. 10.1. There are total of 8 carpals which are joined together and form part of the wrist and base of the hand. The carpal have negligible rotational freedom and is connected to the 5 metacarpals of the thumb and 4 fingers (index, second, third and fourth) by carp metacarpal joints (CMC). The range of movement of the CMC joints cannot be easily measured and it is noted by [5] that only the CMC joints of the thumb, third and fourth finger metacarpals are rotational. The CMC joints of the third and fourth finger metacarpals have small rotational freedom and allows for a flexible palm. The CMC joint of the thumb metacarpal has 3 Degree of Freedom(DOF)s and allows for a dexterous thumb that enable us to grasp and hold objects properly. The metacarpals are in turn connected to the proximal phalanges of the thumb and 4 fingers by the metacarpophalangeal joints (MCP). The MCP joints of the 4 fingers, which are commonly known as the knuckle joints, have much greater rotational freedom than the CMC joints of the 4 fingers and are noted by [5] to have 3 DOFs each. The 4 proximal phalanges of the 4 fingers are connected to 4 middle phalanges by proximal interphalangeal joints (PIP). These 4 middle phalanges are in turn connected to distal phalanges by distal interphalangeal (DIP) joints. The proximal phalanx of the thumb is connected to a distal phalanx by an interphalangeal (IP) joint. All the PIP, DIP and IP joints have 1 DOF each. Should we take the third and fourth finger CMC joints to have 3 DOFs and the MCP joint of the thumb to have 2 DOFs, all of the above results in a total of 27 bones and a total of 32 DOFs [5]. This can make any animator think twice before modeling and animating realistic hands. The individual DOF are as mentioned on Table 10.1.

Figure 10.2 Degrees of Freedom in Hand Model [6]

Joint	Range X	Range Y	Range Z
CMC 1	-180 ~ 20	-180~ 0	-90 ~ 120
CMC 4	-10 ~ 0	0	0~5
CMC 5	-20 ~ 0	0	0 ~ 10
MCP 1	-90 ~ 0	0	0
MCP 2-5	-90 ~ 30	0	-20 ~ 20
PIP 2-5	-120 ~ 0	0	0
DIP 2-5	-80~0	0	0
IP(thumb)	-90 ~ 20	0	0

Table 10.1 DOF of the Joints in Degree (°) [7].

10.2 Modeling and Skinning

Figure 10.3 A 3D Model

To model virtual humans or hands in our case, we decided to make use of 3D Studio Max (3dsMax9) from Autodesk that is easy to use, innovative and professional software for the modeling of 3-Dimensional characters [8]. One of the goals of 3dsMax is to develop an anatomically correct model that has only the necessary number of vertexes and is optimized for animation. By parameter sing a base polygonal mesh, 3dsMax enables a variety of virtual characters to be easily modeled by manipulating parameters through its graphical interface. We modeled a virtual human hand with different fingure segment lengths in 3dsMax.After modeling, the meshes was exported as 3ds object models and imported into PoseRay [9], another famous software tool, and utility to convert 3D model meshes into POV-Ray scenes ( scene description language[10]), Moray(an interactive shareware wireframe modeler[11]) UDO files or Kerkythea meshes [9].

10.2.1. 3D Studio Max

Figure 10.4 3dsMax9 Graphical User Interface(GUI)

Autodesk 3ds Max, formerly 3D Studio MAX, is a modeling, animation and rendering package developed by Autodesk Media and Entertainment. It has modeling capabilities, a flexible plug-in architecture and is able to be used on the Microsoft Windows platform. It can be used by video game developers, TV commercial studios and architectural visualization studios. It is also used for movie effects and movie pre-visualization.

In addition to its modeling and animation tools, the latest version of 3ds Max also features advanced shaders (such as ambient occlusion and subsurface scattering), dynamic simulation, particle systems, radiosity, normal map creation and rendering, global illumination, an intuitive and fully-customizable user interface, and its own scripting language.

10.2.2. Modeling 3D Objects

When Modeling the 3dMax Objects , it is very important to name different parts of the models with relative names, so that it can be uniquely partitioned and identified on java.

Figure 10.5 Named 3D objects on 3dsMax

The above listed names will help to uniquely identify different parts of the full model on java 3D.

10.2.3. PoseRay Composer

PoseRay is a utility to convert 3D model meshes into POV-Ray scenes, Moray UDO files or Kerkythea meshes.

PoseRay can also edit the materials and act as a simple transformation tool for the geometry. PoseRay can also export the modified model to wave front (OBJ) format for use in other 3D programs.

Figure 10.6 PoseRay Composer

Main Features:

· Converts 3D models into full POV-Ray scenes, Moray objects or Wave front models (OBJ+MTL).

· 3D Preview (using the OpenGL ® API) with support for materials, lights and camera

· Smooth and flat surface subdivision

· Material property editor

· Basic geometry transformations

· POV-Ray simulation of environmental illumination from high dynamic range images (HDRI)

· POV-Ray simulation of cartoon rendering

· Support for Poser and DAZ Studio scenes.

· Supported Import Formats:Wavefront(OBJ & MTL),3D studio(3DS), LightWave(LWO), AutoCAD(DXF), POV-Ray 3.X mesh or mesh2, VRML(WRL, GZ), Raw Trian

10.3. Java 3D

Java 3D is a scene graph-based 3D application programming interface (API) for the Java platform. It runs on top of either OpenGL orDirect3D. Since version 1.2, Java 3D has been developed under the Java Community Process [11].

Compared to other solutions, Java 3D is not only a wrapper around these graphics APIs, but an interface that encapsulates the graphics programming using a real, object-oriented concept. Here a scene is constructed using a scene graph that is a representation of the objects that have to be shown. This scene graph is structured as a tree containing several elements that are necessary to display the objects. Additionally, Java 3D offers extensive spatialized sound support.

10.3.1. Java 3D API and Packages

The Java 3D API enables the creation of three-dimensional graphics applications and Internet-based 3D applets. It provides high-level constructs for creating and manipulation 3D geometry and building the structures used in rendering that geometry. With this software, you can efficiently define and render very large virtual worlds.

Packages
com.sun.j3d.audioengines	Provides abstract classes for creating Java 3D audio devices.
com.sun.j3d.audioengines.javasound	Provides a JavaSound-based implementation of a Java 3D audio device.
com.sun.j3d.exp.swing	EXPERIMENTAL: Provides a lightweight JCanvas3D class.
com.sun.j3d.loaders	Provides interfaces and abstract classes for writing Java 3D loaders.
com.sun.j3d.loaders.lw3d	Provides a Java 3D loader for Lightwave 3D scene files.
com.sun.j3d.loaders.objectfile	Provides a Java 3D loader for Wavefront .obj files.
com.sun.j3d.utils.applet	Provides utility classes for running applets as stand-alone applications.
com.sun.j3d.utils.audio	Provides audio utility classes.
com.sun.j3d.utils.behaviors.interpolators	Provides spline-based interpolation behaviors.
com.sun.j3d.utils.behaviors.keyboard	Provides keyboard navigation utility classes.
com.sun.j3d.utils.behaviors.mouse	Provides mouse navigation utility classes.
com.sun.j3d.utils.behaviors.picking	Deprecated: Use com.sun.j3d.utils.pickfast.behaviors instead.
com.sun.j3d.utils.behaviors.sensor	Provides 6DOF sensor behavior classes.
com.sun.j3d.utils.behaviors.vp	Provides ViewPlatform navigation utility classes.
com.sun.j3d.utils.compression	Deprecated: Use com.sun.j3d.utils.geometry.compression instead.
com.sun.j3d.utils.geometry	Provides geometry construction, triangulation, and optimization utility classes.
com.sun.j3d.utils.geometry.compression	Provides compressed geometry utility classes.
com.sun.j3d.utils.image	Provides texture image utility classes.
com.sun.j3d.utils.pickfast	Provides picking utility classes for the new core picking methods.
com.sun.j3d.utils.pickfast.behaviors	Provides picking behaviors for the new core picking methods.
com.sun.j3d.utils.picking	OBSOLETE: provides picking utility classes for the old picking methods.
com.sun.j3d.utils.picking.behaviors	OBSOLETE: provides picking behaviors for the old picking methods.
com.sun.j3d.utils.scenegraph.io	This package provides a Java3D SceneGraph IO capability.
com.sun.j3d.utils.scenegraph.transparency	Provides transparency sorting utility classes.
com.sun.j3d.utils.shader	Provides shader utility classes.
com.sun.j3d.utils.timer	Deprecated: Use java.lang.System.nanoTime() instead.
com.sun.j3d.utils.universe	Provides utility classes for setting up the Java 3D universe, including the viewing configuration.
javax.media.j3d	Provides the core set of classes for the 3D graphics API for the Java platform
javax.vecmath	Provides 3D vector mathematics classes.

Table 10.2 Java3D API Packages

10.3.2. Used java Packages

Following are the list of freely available libraries and packages that are used to design the 3D virtual hand Avatar. Additional Information on those libraries and packages are available at ‘http://download.java.net/media/java3d/javadoc/1.5.2/index.html’.

import com.sun.j3d.loaders.Scene;

import com.sun.j3d.loaders.objectfile.ObjectFile;

import com.sun.j3d.utils.behaviors.mouse.MouseRotate;

import com.sun.j3d.utils.geometry.Box;

import com.sun.j3d.utils.geometry.Sphere;

import com.sun.j3d.utils.image.TextureLoader;

import com.sun.j3d.utils.universe.SimpleUniverse;

import com.sun.j3d.utils.universe.ViewingPlatform;

import java.awt.BorderLayout;

import java.awt.Color;

import java.awt.Dimension;

import java.awt.GraphicsConfiguration;

import java.awt.GridLayout;

import java.awt.Toolkit;

import java.io.FileReader;

import java.io.IOException;

import java.util.Map;

import javax.media.j3d.Alpha;

import javax.media.j3d.Appearance;

import javax.media.j3d.Background;

import javax.media.j3d.BoundingSphere;

import javax.media.j3d.Bounds;

import javax.media.j3d.BranchGroup;

import javax.media.j3d.Canvas3D;

import javax.media.j3d.DirectionalLight;

import javax.media.j3d.Material;

import javax.media.j3d.RotationInterpolator;

import javax.media.j3d.Shape3D;

import javax.media.j3d.Transform3D;

import javax.media.j3d.TransformGroup;

import javax.swing.JFrame;

import javax.swing.JLabel;

import javax.swing.JPanel;

import javax.swing.JSlider;

import javax.swing.event.ChangeEvent;

import javax.swing.event.ChangeListener;

import javax.vecmath.AxisAngle4f;

import javax.vecmath.Color3f;

import javax.vecmath.Vector3f;

10.3.3. java3D Canvas

This is the very first initializing commands needed. There is only single universe exists and the 3D workspace(canvas) is laid on at the ‘CENTER’of the Universe. Graphical User interface is placed on right side boarder.

private SimpleUniverse universe;

private GraphicsConfiguration config;

private Canvas3D canvas;

private void configureCanvas() {

config = SimpleUniverse.getPreferredConfiguration();

canvas = new Canvas3D(config);

canvas.setDoubleBufferEnable(true);

getContentPane().add(canvas, BorderLayout.CENTER);

getContentPane().add(guiPanel(), BorderLayout.EAST);

}

10.3.4. java3D Universe:

Configured canvas is added to the universe.

private Bounds influenceRegion;

private void conigureUniverse() {

universe = new SimpleUniverse(canvas);

universe.getViewingPlatform().setNominalViewingTransform();

}

10.3.5. View Point Camera

This is the actual view seen by the human user. This camera needs to be placed in the same viewing platform which is configured to the prepared canvas. So, campers is looking at the objects that we are about to create on the canvas. Otherwise even the objects are on the canvas, still we can’t see anything.

private void viewPortCamera() {

ViewingPlatform viewPlatform = universe.getViewingPlatform();

TransformGroup viewtransformgroup =

viewPlatform.getMultiTransformGroup().getTransformGroup(0);

Transform3D viewTransform3D = new Transform3D();

viewtransformgroup.getTransform(viewTransform3D);

viewTransform3D.set(new Vector3f(0.0f, 0.60f, 2.8f));

viewtransformgroup.setTransform(viewTransform3D);

}

10.3.6. Adding Lights to Java3D Universe

To see any object, we need to have lights towards that object. Lights are directed at the same influencing region of the objects as well as the canvas. An image file also added to the background.

private void addLightsToUniverse(String bgPic) {

Color3f lightColor = new Color3f(Color.WHITE);

Vector3f lightDirection = new Vector3f(0.0f, 20f, -25f);

DirectionalLight light = new

DirectionalLight(lightColor,lightDirection);

light.setInfluencingBounds(influenceRegion);

root.addChild(light);

TextureLoader textureLoader = new TextureLoader(bgPic, canvas);

Background background = new Background(textureLoader.getImage());

background.setImageScaleMode(Background.SCALE_FIT_ALL);

background.setApplicationBounds(influenceRegion);

root.addChild(background);

}

10.3.7. Adding 3D Model to Canvas

An ‘.obj’ WaveFront file is loaded into java scene and Material for the object is added.3D model is scaled as needed.

TransformGroup model = getHuman(scene, influenceRegion);

Transform3D scaleModel = new Transform3D();

scaleModel.setScale(.8);

Color brown = new Color(255, 128, 64);

Appearance brownAppearance = getAppearance(brown);

namemap.get("p11").setAppearance(brownAppearance);

Separate parts of the 3D model are referenced in java as follows:

joint12 = new TransformGroup();

joint12.addChild(namemap.get("p12"));

Separated parts are joined in Java, using as parent and child nodes of ‘TransformGroup’ Object type.

modifyPartPosition(joint22, new Vector3f(-0.00f, -0.00f, -0.0f), rotate0, joint23);

modifyPartPosition(joint21, new Vector3f(-0.00f, -0.00f, -0.0f),rotate0, joint22);

Figure 10.7 3D Avatar Model on Java3D

10.3.8. Interface Methods and API for Application Integration

Set of public methods are created to make the model to be able to integrate into main GUI application of the Wireless Sign Language Interpreter.

public void setJoint13(int rotation) {

joint13Rot = rotation;

joint13AA.angle = (float) Math.toRadians(joint13Rot);

joint13.getTransform(tmpTrans);

tmpTrans.setRotation(joint13AA);

joint13.setTransform(tmpTrans);

}

References

[1] DEAFSA. Deaf Federation of South Africa, http://www.deafsa.co.za.

[2] M. Huenerfauth, Generating American Sign Language Classifier Predicates

For English-To-ASL Machine Translation, Ph.D. dissertation,

Computer and Information Science, University of Pennsylvania, 2006.

[3] High Quality Flexible H-Anim Hands for Sign

Language Visualisation

Desmond E. van Wyk, James Connan

Department of Computer Science

University of the Western Cape, Private Bag X17 Bellville, 7535, South Africa

[4] The Visual Dictionary. Volume 3: Human body, Bones of The Hand (dorsal view), http://www.infovisual.info/03/027 en.html.

[5] I. Albrecht, “Faces and Hands: Modeling and Animating Anatomical and Photorealistic Models with Regard to the Communicative Competence of Virtual Humans,” Ph.D. issertation, Ph. D. thesis, Universitat des Saarlandes (December, 2005).

[6] Visual Interpretation of Hand Gestures for Human-Computer Interaction: A ReviewVladimir I. Pavlovic,Rajeev Sharma and Thomas S. Huang

transactions on pattern analysis and machine intelligence, vol. 19, no. 7, july 1997

[7] High Quality Flexible H-Anim Hands for Sign Language Visualisation Desmond E. van Wyk, James Connan Department of Computer Science University of the Western Cape, Private Bag X17 Bellville, 7535, South Africa

[8] http://en.wikipedia.org/wiki/3D_studio_max

[9] http://mysite.verizon.net/sfg0000/index.htm

[10] http://www.povray.org/documentation/view/3.6.1/224/

[11] http://www.stmuc.com/moray/

[12] http://en.wikipedia.org/wiki/Java_3D

Abbreviations

MT-Machine Translation

3D - Three Dimensional

CMC-Carp metacarpal joints

DOF-Degree of Freedom

MCP-Mmetacarpo Phalangeal joints

PIP-Proximal Inter Phalangeal joints

DIP-Distal Inter Phalangeal joints

3dsMax-3D Studio Max

GUI-Graphical User Interface

HDRI-High Dynamic Range Images

API-Application Programming Interface