With the rising global energy demand, shale gas and oil emerge as pivotal resources. Recent innovations utilizing CO₂ as an injectant can effectively enhance shale oil and gas recovery and facilitate CO₂ storage within shale reservoirs. However, low-temperature CO₂ injection may result in the coexistence of three hydrocarbon phases, while the abundant nanopores in shale formations also notably influence the phase behavior of reservoir fluids. To optimize shale oil recovery and CO₂ sequestration in shale formations, it is a prerequisite for precisely capturing the effect of confinement on the phase behavior of reservoir fluids within nanopores during CO₂ injection. In this work, we introduce a novel three-phase vapor-liquid-liquid equilibrium calculation algorithm, which is designed to handle the unique phase behavior challenges presented by CO₂ utilization and storage in shale reservoirs. To improve the robustness and efficiency, the proposed algorithm integrates a trust region-based stability test with a hybrid flash calculation algorithm that combines the Newton-Raphson and trust-region methods. Our thermodynamic model incorporates the capillarity effect and shifts in the critical points due to molecule-wall interactions, which are essential for accurate phase behavior simulation under confinement. Initial validations against experimental bulk phase data show promising results, and further investigations indicate that confinement alters three-phase vapor-liquid-liquid equilibria by suppressing two-phase and three-phase regions and shifting boundaries in the phase diagrams. The proposed algorithm not only advances our understanding of multiphase equilibrium in nanoporous media but also enhances the practicality of CO₂ sequestration and improved oil recovery strategies in shale formations.