Design of an experimental scheme for robotic precise visual manipulation based on reinforcement learning

Reinforcement learning (RL) has emerged as a core methodology in intelligent robotics, enabling autonomous agents to interact effectively with dynamic and uncertain environments. Despite its growing importance, RL instruction often remains disconnected from practical engineering applications, which limits students’ ability to translate theoretical knowledge into functional robotic manipulation systems. To address this challenge, this paper presents the design of an experimental scheme for robotic precise visual manipulation based on reinforcement learning and machine vision. The proposed scheme serves both as an instructional platform for deepening theoretical understanding and as a practical environment in which learners can engage with the complete workflow of robotic perception, decision-making, and execution in complex manipulation tasks.

A general-purpose robotic simulation platform was developed using CoppeliaSim as the core environment and integrated with a UR5 robotic manipulator and an RG2 gripper as the execution unit. An RGB-D camera was employed to acquire real-time workspace information, providing synchronized color images and per-pixel depth data, thereby substantially improving robustness in target detection and six-degree-of-freedom pose estimation. Within the perception module, RGB-D inputs were processed through height map generation, image preprocessing, and DenseNet-based feature extraction. The decision module employed a Deep Q-Network (DQN) to evaluate and select pushing and grasping actions. To explicitly connect theoretical instruction with hands-on experimentation, a dual-chain teaching framework was introduced. The theory chain focuses on control fundamentals, perception modeling, and decision optimization, while the practice chain emphasizes scene analysis, simulation-based data collection, and systematic parameter tuning. In addition, a Task Abstraction Layer (TAL) was implemented to facilitate rapid transfer of the learning framework to new manipulation tasks, such as flexible cable assembly, thereby demonstrating the platform’s scalability and generalization capability.

A series of simulation experiments were conducted to assess the effectiveness of the proposed experimental scheme. The results indicate that the robot successfully performed coordinated push-and-grasp operations in unstructured environments, even in the presence of significant object occlusion and scene clutter. Heatmap visualizations of learned Q-values reveal that the DQN-based decision module progressively refined its action selection strategy, converging toward optimal grasping behaviors. Parameter sensitivity analyses further show that both the discount factor and reward weighting exert a strong influence on training performance. A discount factor of 0.7 achieved the best balance between short-term and long-term rewards, yielding stable convergence and a grasping success rate exceeding 90% after 2000 training steps. Modifications to the reward function underscore the need to balance pushing and grasping incentives: eliminating pushing rewards slowed early-stage learning, whereas excessive weighting biased the policy toward pushing actions. Moreover, experiments using the TAL demonstrated that the framework could be adapted to a cable insertion task within approximately three hours, requiring fewer than 50 lines of code changes, thereby confirming its high reusability and adaptability.

The proposed experimental scheme effectively integrates reinforcement learning and machine vision within a hands-on educational framework for robotic manipulation. By emphasizing the explicit relationship between parameter configurations and task performance, the platform enables students to develop intuitive and systematic insights into RL algorithms and decision-making mechanisms. In addition to its instructional value, the scheme demonstrates strong technical feasibility for addressing real-world robotic manipulation challenges, providing a scalable foundation for broader applications. This work offers an innovative approach to bridging theoretical RL education and engineering practice, supporting the development of interdisciplinary talent and the construction of advanced virtual simulation laboratories. Future work will extend the platform to additional manipulation scenarios and control strategies, further strengthening its role in robotics education and research.