Hand Interaction Tutorial

This tutorial will guide you through creating advanced hand interactions using the QWR SDK. You'll learn how to work with hand tracking, gestures, custom hand poses, and physics-based interactions.

Prerequisites

Before starting this tutorial, make sure you have:

• Completed the Quick Start Tutorial
• A device with hand tracking capabilities (e.g., Meta Quest)
• QWR SDK properly installed in your Unity project

Step 1: Create a New Scene

1. Create a new scene (File > New Scene)
2. Save the scene as "HandInteractionDemo"
3. Add a QWR XR Origin prefab to your scene (from Packages/QWR Core/Prefabs/QWR_XROrigin)
4. Add a ground plane (3D Object > Plane) and position it at (0, 0, 0)

Step 2: Create Interactable Objects

Let's create some objects with different interaction behaviors:

Simple Grabbable Cube

1. Create a cube (3D Object > Cube) and position it at (-0.5, 1, 1)
2. Rename it to "GrabbableCube"
3. Add a Rigidbody component
4. Add a QWRInteractable component
5. Configure the QWRInteractable:
   • Grab Type: Both
   • Physics Grab Mode: Kinematic
   • Enable "Snap To Hand"
   • Set Throw Velocity Multiplier to 1.2

Two-Handed Scalable Object

1. Create a cylinder (3D Object > Cylinder) and position it at (0.5, 1, 1)
2. Rename it to "ScalableObject"
3. Add a Rigidbody component
4. Add a QWRInteractable component
5. Configure the QWRInteractable:
   • Grab Type: Both
   • Physics Grab Mode: Kinematic
   • Enable "Two Handed Grab Enabled"
   • Enable "Scale Enabled"
   • Set Min Scale to 0.5
   • Set Max Scale to 2.0

Precision Grab Object

1. Create a sphere (3D Object > Sphere) and position it at (0, 1, 1.5)
2. Rename it to "PrecisionGrabObject"
3. Scale it down to (0.2, 0.2, 0.2)
4. Add a Rigidbody component
5. Add a QWRInteractable component
6. Configure the QWRInteractable:
   • Grab Type: Hand
   • Physics Grab Mode: Kinematic
   • Enable "Precision Grab"

Step 3: Create Custom Grab Points

Let's add custom grab points to an object:

1. Create a new cube (3D Object > Cube) and position it at (0, 1, 0.5)
2. Rename it to "CustomGrabPointsObject"
3. Scale it to (0.5, 0.2, 0.1) to make it look like a handle
4. Add a Rigidbody component
5. Add a QWRInteractable component

Now let's add custom grab points:

1. Create an empty GameObject as a child of CustomGrabPointsObject

2. Name it "LeftGrabPoint"

3. Position it at (-0.2, 0, 0) relative to the parent

4. Add a QWRGrabPoint component

5. Configure the QWRGrabPoint:

   • Hand Type: Left
   • Hand Pose: Grab
   • Enable "Snap Position"
   • Enable "Snap Rotation"

6. Duplicate the LeftGrabPoint

7. Rename the duplicate to "RightGrabPoint"

8. Position it at (0.2, 0, 0) relative to the parent

9. Set its QWRGrabPoint Hand Type to Right

Step 4: Create a Gesture Detection System

Let's create a system that responds to hand gestures:

1. Create a new C# script named "GestureDetector" with the following code:

using UnityEngine;
using QWR.Core.Tracking;
using QWR.Utilities.Logging;

public class GestureDetector : MonoBehaviour
{
    [SerializeField] private GameObject thumbsUpObject;
    [SerializeField] private GameObject pinchObject;
    [SerializeField] private GameObject pointObject;
    
    private QWRHandController leftHand;
    private QWRHandController rightHand;
    
    private void Start()
    {
        // Get hand controllers
        leftHand = QWRHandController.LeftHand;
        rightHand = QWRHandController.RightHand;
        
        // Ensure objects are initially inactive
        if (thumbsUpObject) thumbsUpObject.SetActive(false);
        if (pinchObject) pinchObject.SetActive(false);
        if (pointObject) pointObject.SetActive(false);
        
        // Subscribe to tracking events
        if (leftHand != null)
        {
            leftHand.OnTrackingAcquired += HandleLeftHandTrackingAcquired;
            leftHand.OnTrackingLost += HandleLeftHandTrackingLost;
        }
        
        if (rightHand != null)
        {
            rightHand.OnTrackingAcquired += HandleRightHandTrackingAcquired;
            rightHand.OnTrackingLost += HandleRightHandTrackingLost;
        }
        
        QWRLogger.Log("Gesture Detector initialized");
    }
    
    private void OnDestroy()
    {
        // Unsubscribe from events
        if (leftHand != null)
        {
            leftHand.OnTrackingAcquired -= HandleLeftHandTrackingAcquired;
            leftHand.OnTrackingLost -= HandleLeftHandTrackingLost;
        }
        
        if (rightHand != null)
        {
            rightHand.OnTrackingAcquired -= HandleRightHandTrackingAcquired;
            rightHand.OnTrackingLost -= HandleRightHandTrackingLost;
        }
    }
    
    private void Update()
    {
        CheckGestures();
    }
    
    private void CheckGestures()
    {
        // Check right hand gestures
        if (rightHand != null && rightHand.IsTracked)
        {
            // Thumbs up gesture
            if (thumbsUpObject != null)
            {
                thumbsUpObject.SetActive(rightHand.IsGesture(HandGesture.ThumbsUp));
            }
            
            // Pinch gesture
            if (pinchObject != null)
            {
                pinchObject.SetActive(rightHand.IsGesture(HandGesture.Pinch));
            }
            
            // Point gesture
            if (pointObject != null)
            {
                pointObject.SetActive(rightHand.IsGesture(HandGesture.Point));
            }
        }
    }
    
    private void HandleLeftHandTrackingAcquired(QWRHandController hand)
    {
        QWRLogger.Log("Left hand tracking acquired");
    }
    
    private void HandleLeftHandTrackingLost(QWRHandController hand)
    {
        QWRLogger.Log("Left hand tracking lost");
    }
    
    private void HandleRightHandTrackingAcquired(QWRHandController hand)
    {
        QWRLogger.Log("Right hand tracking acquired");
    }
    
    private void HandleRightHandTrackingLost(QWRHandController hand)
    {
        QWRLogger.Log("Right hand tracking lost");
    }
}

2. Create three spheres in your scene and position them at different locations
3. Name them "ThumbsUpSphere", "PinchSphere", and "PointSphere"
4. Apply different materials to each sphere to distinguish them
5. Create an empty GameObject named "GestureDetector"
6. Add the GestureDetector script to it
7. Assign the three spheres to the corresponding fields in the Inspector

Step 5: Create a Custom Hand Pose

Let's create a custom hand pose for grabbing a specific object:

1. Create a new C# script named "CustomHandPoseObject" with the following code:

using UnityEngine;
using QWR.Core.Interaction;
using QWR.Core.Tracking;

public class CustomHandPoseObject : MonoBehaviour
{
    [SerializeField] private HandPose customPose = HandPose.Grab;
    
    private QWRInteractable interactable;
    
    private void Awake()
    {
        interactable = GetComponent<QWRInteractable>();
        if (interactable == null)
        {
            interactable = gameObject.AddComponent<QWRInteractable>();
        }
        
        // Subscribe to grab events
        interactable.OnGrabbed += HandleGrabbed;
        interactable.OnReleased += HandleReleased;
    }
    
    private void OnDestroy()
    {
        // Unsubscribe from events
        if (interactable != null)
        {
            interactable.OnGrabbed -= HandleGrabbed;
            interactable.OnReleased -= HandleReleased;
        }
    }
    
    private void HandleGrabbed(QWRInteractor interactor)
    {
        // Apply custom pose when grabbed
        QWRHandController handController = interactor.GetComponent<QWRHandController>();
        if (handController != null)
        {
            handController.SetHandPose(customPose);
        }
    }
    
    private void HandleReleased(QWRInteractor interactor)
    {
        // Reset to default pose when released
        QWRHandController handController = interactor.GetComponent<QWRHandController>();
        if (handController != null)
        {
            handController.ResetHandPose();
        }
    }
}

2. Create a new cube in your scene and position it at (0, 1, 2)
3. Rename it to "CustomPoseObject"
4. Add a Rigidbody component
5. Add a QWRInteractable component with standard settings
6. Add the CustomHandPoseObject script
7. In the Inspector, set the Custom Pose to "PinchGrab" (or another pose of your choice)

Step 6: Create a Hand Visualization Controller

Let's create a script to control hand visualization options:

1. Create a new C# script named "HandVisualizationController" with the following code:

using UnityEngine;
using UnityEngine.UI;
using QWR.Core.Tracking;

public class HandVisualizationController : MonoBehaviour
{
    [SerializeField] private Toggle handsVisibleToggle;
    [SerializeField] private Slider transparencySlider;
    [SerializeField] private Dropdown handModelDropdown;
    
    private QWRHandController leftHand;
    private QWRHandController rightHand;
    
    private void Start()
    {
        // Get hand controllers
        leftHand = QWRHandController.LeftHand;
        rightHand = QWRHandController.RightHand;
        
        // Set up UI events
        if (handsVisibleToggle != null)
        {
            handsVisibleToggle.isOn = true;
            handsVisibleToggle.onValueChanged.AddListener(OnHandVisibilityChanged);
        }
        
        if (transparencySlider != null)
        {
            transparencySlider.value = 1.0f;
            transparencySlider.onValueChanged.AddListener(OnTransparencyChanged);
        }
        
        if (handModelDropdown != null)
        {
            handModelDropdown.onValueChanged.AddListener(OnHandModelChanged);
        }
    }
    
    private void OnHandVisibilityChanged(bool visible)
    {
        if (leftHand != null)
            leftHand.SetVisibility(visible);
            
        if (rightHand != null)
            rightHand.SetVisibility(visible);
    }
    
    private void OnTransparencyChanged(float transparency)
    {
        if (leftHand != null)
            leftHand.SetTransparency(transparency);
            
        if (rightHand != null)
            rightHand.SetTransparency(transparency);
    }
    
    private void OnHandModelChanged(int modelIndex)
    {
        if (leftHand != null && rightHand != null)
        {
            leftHand.SetModelDetailLevel(modelIndex);
            rightHand.SetModelDetailLevel(modelIndex);
        }
    }
}

2. Create a UI Canvas in your scene (if you don't already have one)
3. Add a QWRUICanvas component to it
4. Add a Panel to the Canvas
5. Add a Toggle, Slider, and Dropdown to the Panel
6. Label them appropriately ("Hand Visibility", "Transparency", "Model Detail")
7. Create an empty GameObject named "HandVisualizationController"
8. Add the HandVisualizationController script to it
9. Assign the UI elements to the corresponding fields in the Inspector

Step 7: Create a Finger Bend Visualizer

Let's create a visualization of finger bend values:

1. Create a new C# script named "FingerBendVisualizer" with the following code:

using UnityEngine;
using UnityEngine.UI;
using QWR.Core.Tracking;

public class FingerBendVisualizer : MonoBehaviour
{
    [SerializeField] private HandType handType = HandType.Right;
    [SerializeField] private Slider thumbSlider;
    [SerializeField] private Slider indexSlider;
    [SerializeField] private Slider middleSlider;
    [SerializeField] private Slider ringSlider;
    [SerializeField] private Slider pinkySlider;
    
    private QWRHandController handController;
    
    private void Start()
    {
        // Get the appropriate hand controller
        handController = (handType == HandType.Left) ? 
            QWRHandController.LeftHand : QWRHandController.RightHand;
    }
    
    private void Update()
    {
        if (handController != null && handController.IsTracked)
        {
            // Update sliders with finger bend values
            if (thumbSlider != null)
                thumbSlider.value = handController.GetFingerBend(FingerType.Thumb);
                
            if (indexSlider != null)
                indexSlider.value = handController.GetFingerBend(FingerType.Index);
                
            if (middleSlider != null)
                middleSlider.value = handController.GetFingerBend(FingerType.Middle);
                
            if (ringSlider != null)
                ringSlider.value = handController.GetFingerBend(FingerType.Ring);
                
            if (pinkySlider != null)
                pinkySlider.value = handController.GetFingerBend(FingerType.Pinky);
        }
    }
}

2. Create a new UI Panel in your Canvas
3. Add five horizontal Sliders to the Panel, one for each finger
4. Label them appropriately ("Thumb", "Index", "Middle", "Ring", "Pinky")
5. Create an empty GameObject named "FingerBendVisualizer"
6. Add the FingerBendVisualizer script to it
7. Assign the sliders to the corresponding fields in the Inspector

Step 8: Create a Custom Gesture Definition

Let's create a custom gesture that can be detected:

1. Create a new C# script named "CustomGestureDetector" with the following code:

using UnityEngine;
using QWR.Core.Tracking;

public class CustomGestureDetector : MonoBehaviour
{
    [SerializeField] private GameObject targetObject;
    [SerializeField] private HandType handType = HandType.Right;
    
    private QWRHandController handController;
    private QWRGestureDefinition rockSignGesture;
    
    private void Start()
    {
        // Get the appropriate hand controller
        handController = (handType == HandType.Left) ? 
            QWRHandController.LeftHand : QWRHandController.RightHand;
        
        // Create custom gesture definition
        rockSignGesture = ScriptableObject.CreateInstance<QWRGestureDefinition>();
        rockSignGesture.name = "RockSign";
        
        // Configure the gesture (index and pinky extended, others curled)
        rockSignGesture.indexFingerExtendedThreshold = 0.3f;
        rockSignGesture.pinkyFingerExtendedThreshold = 0.3f;
        rockSignGesture.middleFingerExtendedThreshold = 0.7f;
        rockSignGesture.ringFingerExtendedThreshold = 0.7f;
        rockSignGesture.thumbExtendedThreshold = 0.3f;
        
        // Register the custom gesture
        if (handController != null)
        {
            handController.RegisterCustomGesture("RockSign", rockSignGesture);
        }
        
        // Ensure target object is initially inactive
        if (targetObject != null)
        {
            targetObject.SetActive(false);
        }
    }
    
    private void Update()
    {
        if (handController != null && handController.IsTracked && targetObject != null)
        {
            // Check for custom gesture
            bool isRockSign = handController.IsCustomGesture("RockSign");
            targetObject.SetActive(isRockSign);
        }
    }
}

2. Create a new GameObject in your scene (e.g., a sphere with a special material)
3. Position it at an appropriate location
4. Create an empty GameObject named "CustomGestureDetector"
5. Add the CustomGestureDetector script to it
6. Assign the special GameObject to the Target Object field in the Inspector

Step 9: Test Your Scene

1. Save your scene
2. Click the Play button to enter Play Mode
3. Test the various hand interactions:
   • Grab the different objects to see how they behave
   • Try the two-handed scaling with the cylinder
   • Make different gestures to see the corresponding objects appear
   • Use the UI controls to adjust hand visualization
   • Try the custom "rock sign" gesture (index and pinky extended)

Step 10: Build and Deploy

To test on a real device with hand tracking:

1. Go to File > Build Settings
2. Add your current scene to the build
3. Select your target platform (e.g., Android for Quest devices)
4. Click "Switch Platform"
5. Configure platform-specific settings if needed
6. Click "Build" or "Build And Run"

Next Steps

Now that you've learned about advanced hand interactions, you can:

• Explore the Controller Setup Tutorial to learn about controller interactions
• Check out the Multiplayer Setup Tutorial to add networking capabilities
• Review the Hand Tracking Documentation for more detailed information
• Create your own custom hand poses and gestures for specific interactions

Troubleshooting

Hand Tracking Not Working

• Ensure your device supports hand tracking
• Check that hand tracking is enabled in OpenXR settings
• Verify that your hands are within the tracking volume of your device

Custom Gestures Not Detected

• Check the finger bend thresholds in your gesture definition
• Make sure the gesture is properly registered with the hand controller
• Try adjusting the thresholds to make the gesture easier to detect

Hand Visualization Issues

• Verify that the hand model prefabs are correctly assigned
• Check for any errors in the console related to hand visualization
• Make sure the transparency values are in the valid range (0-1)