Cross-modal foundation models are increasingly used for robotic task understanding and planning. However, connecting multimodal observations and natural language instructions to symbolic planners and executable robot actions remains a key challenge in embodied AI. In addressing this challenge, existing methods often rely on structured inputs and lack robustness to language ambiguity, with limited real-world validation. We propose VLA-SP (Vision-Language-Action via Symbolic Planning), a two-stage Embodied Vision-Language-Action framework. VLA-SP consists of a Multi-modal Information Processing Module and a PDDL-based Vision-Language-Action Module, enabling fully automated robotic execution from speech and vision inputs. First, our framework detects task-relevant objects using YOLO and transcribes voice instructions with Faster-Whisper. After obtaining the visual observations and language instructions, the LLM fuses these multimodal inputs and converts them into a planner-verifiable problem.pddl instance under a reusable domain.pddl. Then, Fast Downward validates the generated instance and, upon successful verification, produces a multi-step symbolic plan. If any issues are detected during verification, planner feedback is used for traceable failure diagnosis and iterative self-correction. Once a valid symbolic plan is obtained, the symbolic actions are further grounded into 6-DoF poses and executed through predefined robot skills. Finally, we conduct extensive experiments across seven planning domains in simulation and six real-world manipulation tasks on two robotic platforms, X-Arm and Franka. The experimental results demonstrate that VLA-SP enables reliable execution with high success rates across diverse manipulation tasks. In particular, the complete framework achieves an average success rate of 82% across the six real-world tasks. Results from both simulated and real-world experiments fur-ther demonstrate the strong interpretability, executability, and cross-platform applicability of our framework. Further details are available here.
- Article type
- Year
- Co-author
Open Access
Research Article
Just Accepted
Open Access
Research Article
Online First
Due to the rising demand for Quality of Service (QoS) in the emerging network, this paper considers the routing problem of efficiently planning the transmission path of flow demands to balance the load of the network. Existing classical routing algorithms based on static decision-making are unable to adjust to dynamical scenarios with fluctuating flow. Traditional reinforcement learning-based methods face challenges in large-scale scenarios due to the vast state and action spaces. Besides, the neural networks used for policy generation also suffer from issues like rudimentary feature extraction and limited learning capability. In this paper, we first introduce RequestNet, a novel attention-based model designed to better leverage the information of network topology and feature correlations. We also propose a multi-agent modeling approach for the network packet routing problem, named RequestNet-MA, which treats pairs of edge routers as agents, allowing us to allocate traffic from the perspective of flow demands, thus significantly reducing the dimensions of state and action spaces and enhancing inter-agent communication and cooperation. Extensive experiments and ablation studies demonstrate that the proposed RequestNet-MA can reduce maximum link utilization and scale well to a large-scale network.
Open Access
Online First
To solve collaborative tasks, Multi-Agent Systems (MAS) have been widely used because of their collaboration capabilities and flexibility. They have been applied in various fields, such as robotic collaboration and intelligent transportation systems. In multi-agent collaborative problems, coordinating the collaboration strategies between agents is the most challenging aspect. Traditional Multi-Agent Reinforcement Learning (MARL) methods attempt to transform the team objective into individual objectives for each agent through value decomposition. However, the value allocation mechanisms in existing methods struggle to accurately measure the contribution differences between agents. In this paper, we propose an MARL method based on structured coordination by leveraging the local interaction structure among agents. Depending on the application scenarios, we present explicit and implicit implementation approaches. The explicit method constructs an explicit collaboration graph based on the cooperation relationships between agents, and then uses Shapley values to assess the contribution of each agent in the collaboration graph. Additionally, we propose a method based on self-attention mechanisms to dynamically construct implicit interaction structures. Experimental results demonstrate that, compared to several state-of-the-art multi-agent collaboration algorithms, the two collaboration methods we proposed achieve significant performance advantages across multiple complex collaboration scenarios.
京公网安备11010802044758号