In the last few decades, the particle swarm optimization (PSO) algorithm has been demonstrated to be an effective approach for solving real-world optimization problems. To improve the effectiveness of the PSO algorithm in finding the global best solution for constrained optimization problems, we proposed an improved composite particle swarm optimization algorithm (ICPSO). Based on the optimization principles of the PSO algorithm, in the ICPSO algorithm, we constructed an evolutionary update mechanism for the personal best position population. This mechanism incorporated composite concepts, specifically the integration of the $\epsilon$-constraint, differential evolution (DE) strategy, and feasibility rule. This approach could effectively balance the objective function and constraints, and could improve the ability of local exploitation and global exploration. Experiments on the CEC2006 and CEC2017 benchmark functions and real-world constraint optimization problems from the CEC2020 dataset showed that the ICPSO algorithm could effectively solve complex constrained optimization problems.