As an emerging privacy-preservation machine learning framework, Federated Learning (FL) facilitates different clients to train a shared model collaboratively through exchanging and aggregating model parameters while raw data are kept local and private. When this learning framework is applied to Deep Reinforcement Learning (DRL), the resultant Federated Reinforcement Learning (FRL) can circumvent the heavy data sampling required in conventional DRL and benefit from diversified training data, besides privacy preservation offered by FL. Existing FRL implementations presuppose that clients have compatible tasks which a single global model can cover. In practice, however, clients usually have incompatible (different but still similar) personalized tasks, which we called task shift. It may severely hinder the implementation of FRL for practical applications. In this paper, we propose a Federated Meta Reinforcement Learning (FMRL) framework by integrating Model-Agnostic Meta-Learning (MAML) and FRL. Specifically, we innovatively utilize Proximal Policy Optimization (PPO) to fulfil multi-step local training with a single round of sampling. Moreover, considering the sensitivity of learning rate selection in FRL, we reconstruct the aggregation optimizer with the Federated version of Adam (Fed-Adam) on the server side. The experiments demonstrate that, in different environments, FMRL outperforms other FL methods with high training efficiency brought by Fed-Adam.

Digital twinning and edge computing are attractive solutions to support computing-intensive and service-sensitive Internet of Vehicles applications. Most of the existing Internet of Vehicles service offloading solutions only consider edge–cloud collaboration, but the collaboration between small cell eNodeB (SCeNB) should not be ignored. Service delays far lower than offloading tasks to the cloud can be obtained through reasonable collaborative computing between nodes. The proposed framework realizes and maintains the simulation of collaboration between SCeNB nodes by constructing a digital twin that maintains SCeNB nodes in the central controller, thereby realizing user task offloading positions, sub-channel allocation, and computing resource allocation. Then an algorithm named AUC-AC is proposed, based on the dominant actor–critic network and the auction mechanism. In order to obtain a better command of global information, the convolutional block attention mechanism (CBAM) is used in the digital twin of each SCeNB node to observe its environment and learn strategies. Numerical results show that our experimental scheme is better than several baseline algorithms in terms of service delay.