Chapter 6: Hands and Touch — The Last Bottleneck in Manual-Work Automation

Overview

The hardest moment in manufacturing manual work is often not seeing the object; it is touching it. A pin catches on a hole edge, a thin film sticks to a fingertip, a cable bends inside a connector, a rubber gasket compresses, or a screw crosses a thread. In those moments, vision alone often cannot distinguish progress from failure. Hands, touch, force, compliance, and bimanual coordination sit below GR00T and VLA models, but they decide whether automation works.

This chapter translates dexterous hands, tactile sensing, human-to-robot demonstrations, egocentric human-video pretraining, and action representation into manufacturing-cell language ^[3]; ^[4]; ^[5]; ^[14]; ^[6].

After reading this chapter, you will be able to... - Explain why the bottleneck in manual-work automation is often contact control rather than visual recognition. - Distinguish the data provided by tactile sensors, force sensing, and human demonstrations. - Interpret DexPilot, AnyTeleop, DexUMI, and EgoScale as manufacturing data-collection strategies. - Define the operating KPIs for a first tactile manufacturing cell.

6.1 Why the Hand Is the Last Bottleneck

VLAs are strong at understanding goals and scenes, but they cannot infer every post-contact state from vision. Insertion, alignment, fastening, folding, peeling, wiping, and rework are contact-rich processes. The important signals are often not RGB pixels, but slip, pressure distribution, contact location, force direction, part deformation, and finger compliance.

General manipulation models such as GR00T still operate through a hand and sensor stack ^[1]. That does not mean GR00T has solved tactile manipulation. If the hand cannot hold the part reliably, System 2's plan is irrelevant. If the system lacks touch, System 1 may discover too late, or not at all, whether insertion is progressing, the part is jammed, or the next push will damage the product.

6.2 Tactile Sensors and Human Grasp Data

A tactile sensor is not only a "camera on the fingertip." GelSight and DIGIT-style sensors read elastomer deformation to recover contact geometry and shear cues. Replaceable-skin approaches such as ReSkin address cost and maintenance. Force-torque sensors are coarser than fingertip tactile arrays, but they are strong for insertion and collision detection. The best sensor is determined by the task's failure modes.

Human grasp data is another sensor. DexPilot showed vision-based teleoperation that retargets human hand motion to a robot hand-arm system ^[4]. AnyTeleop proposed a vision-based teleoperation framework across different arm-hand combinations ^[3]. DexUMI uses the human hand as a universal manipulation interface for collecting dexterous robot-hand demonstrations ^[5].

Manufacturers should read this line not as a set of impressive teleoperation devices, but as a data-collection strategy. The system must capture how an expert twists a part slightly, when they reduce force, where they return after sensing failure, and how they support a part temporarily. That data should be linked to task schemas and failure taxonomies before it is used for VLA fine-tuning.

EgoScale extends this line from teleoperation devices to large-scale egocentric pretraining. It pretrains a VLA on more than 20,854 hours of action-labeled egocentric human video and reports average success-rate improvement over a no-pretraining baseline on a 22-DoF dexterous hand ^[14]. In manufacturing terms, expert head-mounted video and hand motion are not just records. They become motor-prior assets for learning which pre-contact and post-contact behaviors predict success.

EgoScale is not a substitute for tactile sensing. Egocentric video captures sequence and visual context around contact, but force magnitude, slip, surface damage, and insertion jams still need separate force/tactile channels and quality logs.

6.3 Reference Humanoids and Tactile Hands

NVIDIA's 2026 Isaac GR00T Reference Humanoid announcement is notable because it includes Sharpa tactile five-finger hands ^[15]. NVIDIA is not defining the humanoid VLA bottleneck as vision and language alone; it is packaging the hand-sensor stack needed to detect and recover from real contact.

The boundary matters. The reference humanoid is not evidence that tactile manufacturing cells have been validated. It is a research integration baseline. A manufacturer should read it as "there is now a shared experimental body with tactile hands," not as "GR00T has solved factory tactile manipulation."

6.4 Limits of Human-to-Robot Transfer

Moving human hand data onto a robot hand is not a coordinate transform. Human hands, Allegro, Shadow, LEAP, SoftHand, and custom grippers differ in degrees of freedom, fingertip size, compliance, friction, and force limits. Retargeting always decides which information to preserve and which information to discard.

In a manufacturing cell, alignment quality matters more than demonstration count. The team must check whether the human motion is feasible for the robot morphology, avoids the fixture, preserves the same tactile event, and stays inside force limits. A failed retargeting run may indicate hardware-task mismatch rather than a model defect.

FAST-style action tokenization matters here ^[6]. Hand motion is high-frequency and continuous, and small changes can alter the result. If the action representation is too long or too coarse, edge latency and policy smoothness fail together. Tactile manipulation requires model architecture and hardware loop rate to be designed together.

6.5 A Tactile Stack for Assembly, Inspection, and Rework

A manufacturing tactile stack is a hierarchy, not a single sensor. The first layer is end-effector choice: parallel gripper, compliant gripper, suction, dexterous hand, or a custom hybrid. The right hand is the one that simplifies the task. The second layer is sensing: wrist force-torque, fingertip tactile, vision, acoustic sensing, and motor current detect different failures. The third layer is policy: diffusion policy, VLA action head, classical impedance control, and scripted recovery may all coexist.

The fourth layer is verification. Even a good tactile signal does not replace final quality inspection. The cell may still require image inspection, electrical test, torque check, or leak test. Tactile policy becomes production automation only when the quality loop is closed.

6.6 Manufacturing Cell Checkpoint

The first goal of a tactile cell is not to show difficult dexterity. It is to detect failure early and recover safely.

6.7 What To Learn Next

Part II has covered the frontier of manipulation through GR00T and VLAs, GEAR agent loops, and hands with touch. Even after the recent GEAR work, the conclusion is the same: large-scale human-video pretraining and world/action models can strengthen pre-contact judgment, but production responsibility requires the tactile, force, and quality loops to close. Part III turns these technologies into factory deployment questions.

References

1. Johan Bjorck et al. (2025). GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv preprint. https://arxiv.org/abs/2503.14734
2. Moo Jin Kim et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model. arXiv preprint. https://arxiv.org/abs/2406.09246
3. Yuzhe Qin et al. (2023). AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System. Robotics: Science and Systems (RSS) 2023. https://www.roboticsproceedings.org/rss19/p015.pdf
4. Ankur Handa et al. (2020). DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System. ICRA 2020. https://doi.org/10.1109/ICRA40945.2020.9197124
5. Mengda Xu et al. (2025). DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation. CoRL 2025 (Best Paper Finalist). https://arxiv.org/abs/2505.21864
6. Karl Pertsch et al. (2025). FAST: Efficient Action Tokenization for Vision-Language-Action Models. arXiv preprint. https://arxiv.org/abs/2501.09747
7. Yi Li et al. (2025). HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation. arXiv preprint. https://arxiv.org/abs/2502.05485
8. Junjie Wen et al. (2024). TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation. arXiv preprint. https://arxiv.org/abs/2409.12514
9. Chen Wang et al. (2025). DexCap: Scalable and Portable Mocap Data Collection for Dexterous Manipulation. arXiv / RSS. https://dex-cap.github.io/
10. Zilin Si et al. (2025). ExoStart: From 10 Exoskeleton Demos to Dexterous Robot Manipulation. arXiv preprint. https://arxiv.org/abs/2506.11775
11. Han Zhang et al. (2025). DOGlove: Dexterous Manipulation with a Low-Cost Open-Source Haptic Force Feedback Glove. RSS 2025. https://arxiv.org/abs/2502.07730
12. Open X-Embodiment Collaboration (2024). Open X-Embodiment: Robotic Learning Datasets and RT-X Models. ICRA 2024. https://arxiv.org/abs/2310.08864
13. Hao-Shu Fang et al. (2025). DEXOP: Passive Exoskeleton for Direct-contact Dexterous Demonstration. arXiv preprint. https://arxiv.org/abs/2509.04441
14. Ruijie Zheng et al. (2026). EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data. arXiv preprint. https://arxiv.org/abs/2602.16710
15. NVIDIA (2026). NVIDIA Announces NVIDIA Isaac GR00T Reference Humanoid Robot for Academic Research. NVIDIA Investor Relations. https://investor.nvidia.com/news/press-release-details/2026/NVIDIA-Announces-NVIDIA-Isaac-GR00T-Reference-Humanoid-Robot-for-Academic-Research/default.aspx