Foundation Model for Robotic Tactile Perception Moves Toward Practical Application with Three Key Technologies

The significant advancements in Vision-Language-Action (VLA) models and world models have dramatically improved a robot’s ability to “see the world.” However, a growing realization is emerging that the real limitation on robotic behavior in practical applications lies not in vision but in tactile interaction itself.

Whether it is plugging and unplugging connectors, opening and closing bottle caps, folding clothes, industrial assembly, household services, or medical care, robots must sense pressure, friction, slippage, and deformation at millisecond intervals during contact and instantly adjust their actions. The success of these tasks requires not only visual information but also the critical contribution of tactile data.

According to a report by 虎嗅網 (Huxiu), the field of robotic tactile research has reached a technological turning point. The development of tactile foundation models hinges on three essential components: the unification of heterogeneous tactile data (FTP-1), the hierarchical reconstruction of tactile roles in robotic policies with high-speed and low-speed control (T-Rex and TouchWorld), and the extension of tactile sensing from dexterous hands to the entire robot body (TACTIC). Tactile perception is gradually evolving from a hardware-dependent sensing capability to an integral part of robotic foundation models.

Breaking the Hardware Barrier: FTP-1

In conventional robotic tactile recognition, tactile perception has often been tightly linked to sensors, leading academia and industry to focus on sensor hardware. However, sensors inherently face engineering challenges such as high cost, fragility, calibration difficulty, and implementation complexity, all of which have hindered progress in robotic tactile perception. Consequently, many researchers have either actively or passively sidestepped the “tactile barrier” in their foundation model studies.

In response, the first general-purpose tactile foundation policy model, FTP-1, jointly developed by Tsinghua University, Sharpa, UC Berkeley, and six other institutions, tackles the core challenge of “sensors” head-on. The central insight behind FTP-1 is straightforward: while general-purpose foundation models in vision (e.g., π0.5) have already demonstrated the capability of large-scale heterogeneous data pretraining combined with downstream task fine-tuning, tactile signals exhibit significant heterogeneity across different hardware. Sensors vary widely in modality (e.g., images, arrays, state vectors), resolution, shape, and contact responses, making unified modeling challenging and the very concept of a tactile foundation model once unthinkable.

FTP-1’s solution lies in designing a Morphology-aware Tactile Token Space (MTTS). It represents 24 functional regions of a robotic hand, each with a token. When encountering different sensors, FTP-1 first groups signals on the sensor by functional region, uses heterogeneous encoders (ViT for images, CNN for tactile arrays, Fourier encoding + MLP for force/torque state vectors), and projects them into a unified space. A shared 300-million-parameter expert tactile Transformer then performs joint modeling. With MTTS, regardless of the sensor’s shape or output signals, data can be translated into a standard “language” that the model understands, allowing FTP-1 to absorb data freely.

FTP-1 aggregates data from 26 sources, covering approximately 3,000 hours of tactile operation data across 21 sensor types, including human teleoperation and robot demonstrations. After pretraining, the model was fine-tuned and evaluated on downstream tasks across five hardware configurations from five institutions, covering 14 tasks, achieving cross-sensor tactile model evaluation.

FTP-1 achieved an average success rate of 62.5% in fine-tuned tasks with sensor types seen during pretraining, surpassing π0.5’s 45.3%. More importantly, FTP-1 achieved a 46.6% success rate for two sensor types unseen during pretraining, significantly outperforming π0.5’s 15.0%. Ablation experiments confirmed that FTP-1’s gains stem from the transferable tactile knowledge learned during pretraining. By translating diverse tactile hardware into a unified “tactile language,” FTP-1 is the first tactile model to unlock the potential of large-scale pretraining.

Separating High-Speed and Low-Speed

Control: T-Rex

When a human grabs a cup and it starts to slip, their fingers reflexively tighten without conscious thought. This millisecond-level tactile reflex, bypassing cognitive processes, is a significant challenge for robots.

Traditional robotic strategies, from Robot Diffusion Policies (RDP) to Tactile-VLA, have either treated tactile signals as mere input tokens processed within a single model—resulting in competition between low-speed planning and high-speed reflexes within the same inference loop—or employed independent “low-speed” latent diffusion strategies and “high-speed” tactile controllers. However, RDP models were typically trained for parallel grippers and lacked scalability to high-degree-of-freedom dexterous hands.

In contrast, T-Rex, developed in June 2026 by UC Berkeley, Stanford University, and NVIDIA, and TouchWorld, introduced in July 2026 by Harbin Institute of Technology and PoXiao Intelligence, demonstrate the necessity of separating high-frequency tactile feedback and low-frequency semantic planning through “asynchronous processing within a unified model” and “hierarchical prediction and correction,” respectively.

T-Rex’s core innovation lies in its Mixture of Experts (MoE) architecture, where latent experts predict future visual representations, motion experts handle low-frequency (5 Hz) flow-matching denoising, and tactile experts perform high-frequency (20 Hz) tactile optimization. Unlike RDP, which encodes actions with an action tokenizer, conducts low-frequency planning in a latent space using diffusion strategies, and relies on separate tactile controllers for high-frequency corrections, T-Rex employs Asynchronous Cascaded Flow Matching. This integrates high- and low-frequency processing into the same generative process.

Specifically, T-Rex splits the 10-step flow-matching denoising process (the motion generation phase) into two parts. Motion experts combine visual and language information to plan globally in the first six steps. Visual-language model KV caches are then frozen, allowing tactile experts to refine the remaining four steps and optimize actions swiftly. This maintains knowledge sharing within a unified model while separating high- and low-frequency information.

During execution, motion experts perform a single 16-step low-frequency visual-language inference, while tactile experts execute only lightweight, four-step denoising for corrections based on tactile information. This not only reduces computational requirements by avoiding repeated visual feature recalculations but also increases the frequency of tactile feedback.

T-Rex demonstrated a 30% higher average success rate than the strongest baseline (EgoScale) across 12 dexterous tactile manipulation tasks, highlighting its superior precision in force control and millisecond-level tactile response. Ablation experiments revealed a 5% drop in success rates when synchronous mode was used, confirming that asynchronous scheduling itself contributes to performance gains.

Three-Layer World Model: TouchWorld

While T-Rex separates high-speed and low-speed control through asynchronous scheduling within a single model, TouchWorld explicitly separates three temporal scales using a hierarchical architecture. TouchWorld emphasizes the temporal separation of tactile feedback and its prediction.

TouchWorld divides its world model architecture into three layers: the high-level planning layer (1 Hz), which uses Qwen3-VL-4B to decompose long-term instructions into actionable subtasks and Wan2.2-TI2V-5B as a tactile world model to predict future visual-tactile sub-goals; the mid-level visual-tactile policy layer (10 Hz), where a diffusion Transformer generates nominal motion blocks based on subtask instructions and predicted tactile goals; and the high-frequency tactile condition optimization layer (30 Hz), where a Tactile Residual Transformer (TRT) refines actions using the latest tactile history during execution gaps.

Unlike T-Rex, which disperses the capabilities of a tactile world model across three future visual representation experts within a unified Mixture of Transformers (MoT), TouchWorld fuses tactile perception into an independent, explicit generative world model that outputs visual-tactile sub-goal images directly usable by downstream policies.

TouchWorld demonstrated impressive performance in six real-world tasks, achieving an average success rate of 65.0% in clean settings—15.7 percentage points higher than the strongest baseline (FTP-1, 49.3%).

Prospects for Tactile Foundation Models

The three technologies—FTP-1, T-Rex, and TouchWorld—address fundamental challenges in robotic tactile perception from different angles. FTP-1 provides a foundation for unifying data across heterogeneous sensors, while T-Rex and TouchWorld illustrate how to utilize tactile feedback in practical robotic control.

These studies indicate that tactile perception is evolving from merely processing sensor outputs to becoming a core component of robotic foundation models. The paradigm of “large-scale data + foundation models,” established in the vision domain, is finally becoming applicable to tactile perception.

The cross-sensor transfer capabilities of FTP-1, in particular, hold the potential to alleviate issues related to tactile sensor production costs and compatibility. A general-purpose model independent of specific sensors would grant manufacturers greater freedom in sensor selection, fostering price competition and quality improvement.

The hierarchical control architectures demonstrated by T-Rex and TouchWorld have broad applications, from industrial robots to service robots. Fields that demand precise force control, such as assembly processes, logistics involving uncertain objects, and caregiving environments involving human contact, stand to benefit significantly from high-speed tactile feedback.

Considering the ongoing advancements in TACTIC (expanding tactile sensing from dexterous hands to the entire robot body), robots may eventually utilize full-body tactile information to interact with their surroundings. By integrating tactile information, robotic behavior planning—previously overly reliant on vision—could evolve to become more robust and adaptive.

Editorial Opinion

In the short term, the establishment of cross-sensor transfer learning by FTP-1 may bring competition to the tactile sensor market. Currently, manufacturers often face lock-in to specific sensors, but widespread adoption of unified models could increase sensor options, spurring a virtuous cycle of lower prices and higher quality. Progress toward standardizing open tactile datasets based on FTP-1 is expected in the next three to six months.

From a long-term perspective, the establishment of tactile foundation models is likely to influence the architecture of robotics as a whole. With the integration of tactile perception into current VLA models that predominantly rely on vision and language, more human-like multimodal behavior generation could become a reality. Within one to three years, these technologies might reach commercial viability in precision assembly and caregiving sectors that require tactile feedback.

However, challenges remain before tactile foundation models can be fully integrated into practical robots, including model optimization for real-time inference, lightweight designs for edge devices, and cost considerations. Just as T-Rex improved computational efficiency with its asynchronous architecture, developing lightweight models capable of executing tactile processing on edge devices will likely become the next major focus.

References

• ” 新一轮触觉热，真正值得关注的不只是硬件 ”, by 42号电波© — 虎嗅网, 2026-07-15T10:15:09.000Z (ARR)
• Source URL: https://www.huxiu.com/article/4875532.html?f=rss