Trajectory Optimization for In-Hand Manipulation with Tactile Force Control

The strength of the human hand lies in its ability to manipulate small objects precisely and robustly. In contrast, simple robotic grippers have low dexterity and fail to handle small objects effectively. This is why many automation tasks remain unsolved by robots. This paper presents an optimization-based framework for in-hand manipulation with a robotic hand equipped with compact Magnetic Tactile Sensors (MTSs). The small form factor of the robotic hand from Shadow Robot introduces challenges in estimating the state of the object while satisfying contact constraints. To address this, we formulate a trajectory optimization problem using Nonlinear Programming (NLP) for finger movements while ensuring contact points to change along the geometry of the fingers. Using the optimized trajectory from the solver, we implement and test an open-loop controller for rolling motion. To further enhance robustness and accuracy, we introduce a force controller for the fingers and a state estimator for the object utilizing MTSs. The proposed framework is validated through comparative experiments, showing that incorporating the force control with compliance consideration improves the accuracy and robustness of the rolling motion. Rolling an object with the force controller is 30% more likely to succeed than running an open-loop controller. The demonstration video is available at https://youtu.be/6J_muL_AyE8.

💡 Research Summary

The paper presents a complete framework for achieving precise in‑hand manipulation—specifically, rolling a small cylindrical object between two fingers—using a compact, human‑like Shadow robotic hand equipped with Magnetic Tactile Sensors (MTS). The authors first formulate a nonlinear trajectory optimization problem that simultaneously plans finger joint torques and the evolution of contact points along the curved finger surfaces. By introducing a virtual prismatic joint for the thumb and enforcing a no‑slip constraint (Eq. 4‑5), the optimizer guarantees that the contact points move continuously rather than remaining fixed at the fingertip, which is essential for natural rolling. The optimization minimizes a weighted quadratic cost on state error, control effort, and a distance term that forces the fingers to stay in contact, while satisfying static equilibrium, friction cone, and contact‑point continuity constraints. The problem is solved with IPOPT via CasADi, yielding a reference trajectory of joint torques and contact forces.

Because the hand lacks visual feedback, the authors develop a state estimator based solely on the 17 MTS mounted on the fingers. Each sensor provides a magnetic field magnitude proportional to normal force. The raw magnitudes are normalized, the strongest sensor is identified, and a weighted average of the nearest‑neighbor sensor positions yields an estimated contact point (Eq. 9). By tracking the displacement of this point relative to the object geometry, the estimator reconstructs the object’s planar pose (center position and orientation) in real time. Although the estimate exhibits noise, the multi‑sensor averaging significantly reduces error compared with using a single sensor.

To close the loop, a hierarchical finger controller is designed. A position‑error PID generates a reference fingertip velocity, which is then tracked by a Jacobian‑based velocity controller. Since the Shadow hand’s tendons couple multiple joints, direct inverse kinematics is infeasible; instead, a pseudo‑inverse of the Jacobian computes joint‑velocity commands, constrained by a linear least‑squares problem (Eq. 14) that keeps joint updates within a small bound ε. Pure position/velocity control, however, cannot guarantee stable grasp during rolling because external disturbances cause large force variations. Therefore, a force controller is added: the measured contact forces from the MTS are compared to desired forces, and corrective torques are injected. The authors also experimentally identify the tendon stiffness of each finger, allowing the controller to compensate for compliance and avoid excessive joint deflection.

The experimental validation consists of three control configurations: (1) open‑loop execution of the optimized trajectory, (2) open‑loop plus the force controller, and (3) the full framework including the MTS‑based state estimator. In all cases the task is to roll the cylinder 180° between the first finger (4 DoF) and the thumb (5 DoF). Results show that adding the force controller raises the success rate by roughly 30 % (from ~70 % to near‑100 %), reduces the average orientation error from 0.8 rad to 0.3 rad, and yields smoother contact‑force profiles. The estimator further improves performance by providing accurate pose feedback, which allows the force controller to react appropriately to deviations.

Key contributions of the work are: (i) a contact‑dynamic trajectory optimization that respects continuous contact‑point motion, (ii) a compact magnetic tactile‑sensor‑based pose estimator that operates without vision, (iii) a compliant‑aware force controller tailored to tendon‑driven hands, and (iv) a thorough experimental demonstration on a real human‑like robotic hand. The authors argue that these elements together enable dexterous in‑hand manipulation comparable to human capability, even with a small, lightweight hand. Future directions include extending the method to three‑dimensional objects, multi‑finger (more than two) manipulation, and integrating learning‑based components to handle more complex, unstructured contact scenarios.