This paper investigates the joint resource allocation problem in Reconfigurable Intelligent Surface (RIS)-assisted cooperative non-orthogonal multiple access device-to-device (CNOMA-D2D) cellular networks. To tackle the high-dimensional non-convex joint optimization of power control, RIS phase configuration and channel assignment, we propose an integrated user pairing strategy, PIP-UP, quantifying utility through factors, phase alignment, interference suppression and power difference, neglected in existing methods. Furthermore, we develop a hybrid deep reinforcement learning algorithm, A3TD, combining the parallel exploration capability of Asynchronous Advantage Actor-Critic (A3C) with the stable continuous optimization of Twin Delayed Deep Deterministic Policy Gradient (TD3). This integration enables efficient and robust joint optimization of D2D channel allocation, transmit power, and RIS phase shifts. Simulation results demonstrate that the proposed A3TD algorithm significantly outperforms baseline algorithms, Actor-Critic (AC), Deep Deterministic Policy Gradient (DDPG) and TD3, in terms of sum rate and convergence speed, validating its effectiveness for resource management in complex RIS-assisted CNOMA-D2D networks.