Skip to main content

Hand Tracking

QWR SDK provides advanced hand tracking capabilities with realistic hand models, animations, and a comprehensive gesture recognition system. This page explains how to use and customize the hand tracking features.

Hand Tracking Overview​

The hand tracking system in QWR SDK includes:

  • Realistic hand models with natural animations
  • Finger tracking with accurate joint positions
  • Gesture and pose recognition
  • Physics-based interactions
  • Haptic feedback simulation
  • Cross-platform compatibility

QWRHandController​

The QWRHandController component is the central element of the hand tracking system. It manages hand visualization, tracking, and interactions.

Accessing Hand Controllers​

You can access the hand controllers through static properties:

// Get references to hand controllers
QWRHandController leftHand = QWRHandController.LeftHand;
QWRHandController rightHand = QWRHandController.RightHand;

// Or find them in the scene
QWRHandController[] handControllers = FindObjectsOfType<QWRHandController>();

Key Properties and Methods​

// Check if hand is currently tracked
bool isTracked = leftHand.IsTracked;

// Check if hand is currently grabbing
bool isGrabbing = leftHand.IsGrabbing;

// Get current hand pose
HandPose currentPose = leftHand.CurrentPose;

// Get finger bend values (0-1 range)
float indexBend = leftHand.GetFingerBend(FingerType.Index);

// Get finger tip position
Vector3 indexTipPos = leftHand.GetFingerTipPosition(FingerType.Index);

// Force haptic feedback
leftHand.TriggerHapticFeedback(amplitude: 0.5f, duration: 0.1f);

// Set hand visibility
leftHand.SetVisibility(visible: true);

Hand Events​

The QWRHandController provides several events you can subscribe to:

// Subscribe to tracking events
leftHand.OnTrackingAcquired += HandleTrackingAcquired;
leftHand.OnTrackingLost += HandleTrackingLost;

// Subscribe to grab events
leftHand.OnGrabBegin += HandleGrabBegin;
leftHand.OnGrabEnd += HandleGrabEnd;

// Subscribe to pose events
leftHand.OnPoseChanged += HandlePoseChanged;

// Example event handler
private void HandleGrabBegin(QWRHandController hand, QWRInteractable grabbedObject)
{
    Debug.Log($"Hand {hand.HandType} grabbed {grabbedObject.name}");
}

Hand Models and Visualization​

QWR SDK includes realistic hand models with animations for different poses and interactions.

Hand Model Structure​

Each hand model includes:

  • Skeletal mesh with proper bone hierarchy
  • Materials with skin shaders
  • Animation controller for different poses
  • Colliders for physics interactions

Customizing Hand Appearance​

You can customize the appearance of hands:

  1. In the scene hierarchy, select QWR XR Origin > Camera Offset > Left/Right Hand Controller
  2. In the Inspector, find the QWRHandVisualizer component
  3. Modify properties such as:
    • Hand Material: Change the material used for the hand
    • Hand Scale: Adjust the overall size of the hand
    • Finger Lengths: Fine-tune individual finger proportions

Using Custom Hand Models​

To use your own hand models:

  1. Create a hand model prefab with the same bone hierarchy as the default model
  2. In the scene hierarchy, select the Hand Controller
  3. In the Inspector, find the QWRHandController component
  4. Replace the Hand Model Prefab with your custom model
caution

Custom hand models must follow the same bone naming convention as the default models for animations to work correctly.

Gesture Recognition​

QWR SDK includes a powerful gesture recognition system that can identify common hand poses and custom gestures.

Built-in Gestures​

The SDK recognizes several built-in gestures:

  • Point (index finger extended)
  • Pinch (thumb and index finger touching)
  • Grab (fingers curled in grabbing position)
  • Flat hand (all fingers extended)
  • Fist (all fingers curled)
  • Thumbs up
  • OK sign

Detecting Gestures​

// Check if a specific gesture is active
bool isPinching = leftHand.IsGesture(HandGesture.Pinch);
bool isPointing = leftHand.IsGesture(HandGesture.Point);

// Get the current active gesture
HandGesture currentGesture = leftHand.CurrentGesture;

// Subscribe to gesture events
leftHand.OnGestureBegin += HandleGestureBegin;
leftHand.OnGestureEnd += HandleGestureEnd;

// Example event handler
private void HandleGestureBegin(QWRHandController hand, HandGesture gesture)
{
    if (gesture == HandGesture.ThumbsUp)
    {
        Debug.Log("Thumbs up gesture detected!");
    }
}

Creating Custom Gestures​

You can define custom gestures using the QWRGestureDefinition scriptable object:

  1. In the Project window, right-click and select Create > QWR > Gesture Definition
  2. Configure finger bend thresholds for each finger
  3. Assign the gesture definition to the QWRHandController component
// Create a custom gesture definition in code
QWRGestureDefinition customGesture = ScriptableObject.CreateInstance<QWRGestureDefinition>();
customGesture.name = "Rock Sign";
customGesture.indexFingerExtendedThreshold = 0.3f;
customGesture.pinkyFingerExtendedThreshold = 0.3f;
customGesture.middleFingerExtendedThreshold = 0.7f;
customGesture.ringFingerExtendedThreshold = 0.7f;
customGesture.thumbExtendedThreshold = 0.3f;

// Register the custom gesture
QWRHandController.LeftHand.RegisterCustomGesture("RockSign", customGesture);

Physics Interactions​

The hand tracking system includes physics-based interactions with objects in the scene.

Hand Physics​

Each finger has colliders that can interact with physics objects:

  • Finger tips have sphere colliders for precise interactions
  • Palm has a box collider for larger surface interactions
  • Physics material properties can be adjusted for different friction levels

Configuring Physics Interactions​

You can adjust physics interaction settings:

  1. In the scene hierarchy, select the Hand Controller
  2. In the Inspector, find the QWRHandPhysics component
  3. Modify properties such as:
    • Collision Layer Mask: Define which layers the hand can interact with
    • Physics Update Rate: Adjust the frequency of physics updates
    • Finger Stiffness: Control how rigid the fingers are during interactions

Performance Considerations​

Hand tracking can be resource-intensive. Here are some optimization tips:

  • Use the LOD (Level of Detail) system for hand models
  • Adjust physics update rate based on your target platform
  • Disable visual effects like finger shadows on lower-end devices
  • Use the simplified hand model option for better performance
// Set hand model detail level (0-2, where 0 is highest detail)
leftHand.SetModelDetailLevel(1);

// Enable/disable physics interactions
leftHand.EnablePhysics(enabled: false);

// Enable/disable visual effects
leftHand.EnableVisualEffects(enabled: false);

Platform-Specific Considerations​

Different XR platforms have varying levels of hand tracking support:

  • Quest/Meta Devices: Full hand tracking support with high fidelity
  • Windows Mixed Reality: Basic hand tracking with limited gesture support
  • SteamVR: Varies by headset, may require additional configuration
  • Mobile AR: Limited support, primarily through front-facing camera

Troubleshooting​

Hand Tracking Not Working​

If hand tracking isn't working:

  • Ensure your device supports hand tracking
  • Check that hand tracking is enabled in OpenXR settings
  • Verify that the correct tracking mode is selected
  • Make sure there's sufficient lighting for camera-based tracking

Hand Model Visualization Issues​

If hand models don't appear correctly:

  • Check that the hand model prefabs are correctly assigned
  • Verify that the hand scale settings are appropriate
  • Ensure the materials are properly configured
  • Check for animation controller issues

Gesture Recognition Problems​

If gestures aren't being recognized:

  • Adjust the gesture recognition thresholds
  • Ensure your hand is within the tracking volume
  • Check for conflicts between similar gestures
  • Calibrate the hand tracking system for your specific hands

Next Steps​

Now that you understand the hand tracking system, you can explore: