In Federated Learning (FL), the aggregation servers inevitably face attacks from malicious clients because they cannot directly constrain the behaviour of clients. Effectively detecting and mitigating the impact of these malicious clients on the global model is a critical challenge. This paper proposes a secure aggregation algorithm, named Ordering Points To Identify the Clustering Structure (OPTICS) Clustering Algorithm and Client Selection Federated Learning (OCACSFL), designed to identify and exclude malicious clients. By employing an improved OPTICS clustering algorithm, OCACSFL enables the aggregation server to effectively distinguish between benign and malicious clients, ensuring that malicious updates do not contribute to the global model aggregation. If a client is detected to upload malicious updates multiple times, OCACSFL permanently rejects that client’s updates. Experimental results demonstrate that OCACSFL can efficiently detect and exclude malicious clients across various datasets and machine learning models, showcasing exceptional performance and robustness.