Symmetry-Aware Visuo-Tactile Fusion via Bilateral Force Priors

Overview

TL;DR — We propose a Cross-Modal Transformer with physics-informed symmetry regularization for visuo-tactile fusion. It achieves 96.59% insertion success on the TacSL benchmark, nearly matching privileged force sensing while using only realistic tactile inputs.

Contact-rich tasks like plug insertion break down when you rely on vision alone — close-range alignment demands force-level precision that cameras simply cannot provide. Tactile sensing fills that gap, but naïve fusion of vision and touch often hurts more than it helps. The core issue is that raw concatenation dilutes the already-weak tactile signal.

We address this with two ideas: a Cross-Modal Transformer (CMT) that learns structured alignment between visual and tactile features through self- and cross-attention, and a bilateral symmetry loss grounded in Newton’s Third Law — if both fingers grasp the same object, their tactile embeddings should mirror each other. This simple physics-informed constraint stabilizes training without any extra labels.

Comparison of vision-only, tactile-only, and visuo-tactile fusion for plug insertion — Vision gives you the global picture but loses precision at contact. Tactile sensing captures rich contact forces but lacks spatial context. Our fusion combines both for balanced, robust insertion.

Method

Contributions

Cross-Modal Transformer (CMT) — Structured visuo-tactile fusion via hierarchical self- and cross-attention, replacing naïve concatenation with learned inter-modal alignment.
Physics-informed symmetry regularization — Bilateral force-balance constraint inspired by Newton’s Third Law. Vertically flipping the right tactile image and enforcing embedding consistency via MSE stabilizes representations with zero extra annotation cost.
State-of-the-art insertion performance — 96.59% success on TacSL, surpassing all fusion baselines and nearly matching privileged wrist + contact-force sensing (96.09%).

Architecture: Cross-Modal Transformer fusion vs naive and gated baselines — Overall architecture. The policy takes robot states, a wrist image, and left/right tactile inputs. The fusion module is instantiated as (a) naïve early fusion, (b) linear gated fusion, or (c) our Cross-Modal Transformer with structured self- and cross-attention.

Bilateral symmetry regularization diagram — Symmetry regularization: the right tactile image is vertically flipped before encoding, and an MSE loss between left and flipped-right embeddings enforces bilateral force consistency.

Results

96.59%

Insertion success rate
CMT + symmetry reg.

153 fps

Inference speed
real-time capable

100%

Screw-task success
tactile only

Method	Privileged	Reduced	Vision	Tactile	Succ. Rate (%)
Privileged	✓				96.74
+ Contact Forces	✓				98.96 (+2.22)
Tactile		✓		✓	91.41
Vision		✓	✓		93.23
Vision + Contact Forces		✓	✓		96.09 (+2.86)
Fusion – Naive		✓	✓	✓	92.97
Fusion – Gated		✓	✓	✓	94.53 (+1.56)
Fusion – CMT (ours)		✓	✓	✓	96.22 (+3.25)
Fusion – Gated + Symmetry Reg.		✓	✓	✓	95.05 (+2.08)
Fusion – CMT + Symmetry Reg. (ours)		✓	✓	✓	96.59 (+3.62)

On the TacSL benchmark, CMT with symmetry regularization outperforms naïve early fusion (92.97%) and linear gated fusion (94.53%) by a clear margin, and nearly matches the privileged wrist + contact-force setting (96.09%) — despite using only realistic tactile images as input.

Adding tactile sensing to vision consistently improves success rates by +2.2% to +2.8% across all fusion strategies, confirming that touch provides genuinely complementary information for precise alignment. Tactile-only policies already reach 91.41%, and on the screw insertion task tactile alone hits a perfect 100%.

Qualitative comparison: tactile force fields during insertion — Tactile force fields during plug insertion. Naïve fusion (top) produces unbalanced, noisy contacts. Our method (bottom) yields symmetric, consistent force distributions across all insertion phases.

Citation

@inproceedings{lee2026symmetry,
  title     = {Symmetry-Aware Fusion of Vision and Tactile Sensing
               via Bilateral Force Priors for Robotic Manipulation},
  author    = {Lee, Wonju and Grimaldi, Matteo and Yu, Tao},
  booktitle = {IEEE International Conference on Robotics and
               Automation (ICRA)},
  year      = {2026}
}

Overview

Method

Contributions

Results

Video

Citation