Doing It Right with FiftyOne: Sim-to-Real Gesture Classification for Human-Robot Interaction

• ~200 real clips beat ~2,000 synthetic ones. On real robot test footage, a gesture classifier trained on 204 real clips scores .760 accuracy against .590 for one trained on 1,980 synthetic clips. In-domain data gets in-domain performance.
• Data changes moved the needle, model architecture did not. In this sim-to-real gesture classifier, pose normalization added +.17, windowing added +.17, real training data added +.17, and longer windows added +.069. Swapping LSTM for a temporal convolutional network was worth +.005, less than the .017 spread from re-running one model at a different random seed.
• The sim-to-real gap is not one gap, it is five. This project found feature-space separation, apparent person scale, a labeling convention mismatch between two annotating teams (roughly 40% of the apparent gap), the pose estimator's perspective sensitivity, and a residual 14-point performance gap.
• Aggregate classification metrics hide per-class nuance. There is no "51% accurate gesture model" here. Three gestures are essentially solved, three sit at or below the 14.3% chance floor, and the same bimodality shows up in Steps 3 and 5.
• Zero-shot on in-the-wild robot footage more than doubled. The rebuilt classifier scores 51.3% on 840 clips of the author's own robot footage it never trained on, against 24% in the 2024 version that had that footage in its training set.

Why gesture classification for human-robot interaction?

• …gesture classification results are around random chance level…
• Does gesture recognition accuracy improve in different scenes?

Step 1: Loading the RoCoG-v2 data

Step 2: Adding pose prelabels and embeddings

Step 2.5: Normalizing the pose features

Step 3: Classifying the gesture video clips

• Advance, FollowMe, MoveInReverse, and Rally all have F1 scores above .5
• Attention, Halt, and MoveForward have F1 scores at or below ~.1

What this tells us

Step 4: Temporal classification with a causal model you can deploy

• Gestures in the synthetic samples start partway through the clip. The beginning and end frames are specified in a separate metadata file.
• Gestures in the real samples also start partway through the clip. However, the beginning and end frames aren’t specified, only the overall label.

Finding a common representation

• The gesture begins when the wrist keypoint goes above the elbow keypoint for the first time in the clip (minus a learned offset*)
• The gesture ends when the wrist keypoint goes below the elbow keypoint for the last time in the clip (plus a learned offset*)

Step 5: Model training and evaluation

• Adding real data to the training set gives a minor degradation to the synthetic evaluation metrics, but a ~20 point improvement to eval metrics on the real dataset
  • The bottom row reveals an asymmetry between sim and real data: training on 204 real clips transfers to real and not at all to synthetic; training on 1,980 synthetic clips transfers partway to both.
• Window length is the most important model parameter. Across both LSTM and TCN architectures, accuracy improves monotonically by ~7 percentage points as the temporal window length is extended from 0.5 to 2.0 seconds.
• Model architecture is significantly less important; accuracy varies by about .005 between different model architectures, less than the ~.017 variance from choosing a different random seed.
• When trained on synthetic+real, the TCN (2.0s window) model’s clip classification accuracy on real-test is 82%, and the LSTM’s (2.0s window) accuracy is 81%; both modest improvements over the 73% reported by the RoCoG paper’s authors.
• Models trained on real-train (n=204) outscored models trained on sim-train (n=1980) when evaluated on real-test (n=100).
• The average values hide the per-class discrepancies: adding real training data improves per-class performance, not new model architectures.
• Temporal evaluation in the last table is particularly eye-opening: 204 real training clips (0.2517) essentially tie 1,980 synthetic + 204 real (0.2542), and both beat synthetic alone (0.1497). At this scale the real clips drive the performance.

Step 6: Zero-shot evaluation on in-the-wild robot data

• Headline accuracy improved from 24% in my 2024 paper to 51.3% in this iteration, a > 2X improvement. Notable because the 2024 version included the in-the-wild data in training, and this time it trained only on RoCoG.
• Once again, the per-class table and eval runs reveal nuance around the overall metrics:
  • Advance has no valid temporal predictions in this split - hence the mAP value could not be computed.
  • FollowMe, MoveForward and Rally are near-perfectly accurate and have temporal mAP > 0.5
  • Attention, Halt, and MoveInReverse are effectively impossible to classify or localize temporally - at or below chance level.

What actually closed the sim-to-real gap

References

Appendix: YOLO-pose prelabeling code snippets

import cv2
import fiftyone as fo
import fiftyone.core.storage as fos
from ultralytics import YOLO

model = YOLO("yolo11n-pose.pt")

for sample in dataset.iter_samples(autosave=True, progress=True):
    # Pull locally if media stored in the cloud
    with fos.LocalFile(sample.filepath, mode="r") as local_path:
        cap = cv2.VideoCapture(local_path)
        frame_number = 0
        while True:
            ok, frame = cap.read()
            if not ok:
                break
            frame_number += 1
            h, w = frame.shape[:2]

            result = model(frame, verbose=False)[0]
            if result.keykeypoints.xy) == 0:
                continue

            # Most-confident person: RoCoG clips have exactly one subject.
            i = int(resul
            if float(result.boxes.conf[i]) < 0.25:
                continue

            sample.frames fo.Keypoints(
                keypoints=[fo.Keypoint(
                    label="person",
                    # Fifed to [0, 1]
                    points=[(x / w, y / h)
                            for x, y in result.keypoints.xy[i].tolist()],
                    confidence=result.keypoints.conf[i].tolist(),
                )]
            )
        cap.release()

dataset.skeletons = {"keypoints": fo.KeypointSkeleton(
    labels=COCO17_LABELS, edges=COCO17_EDGES,
)}
dataset.save()