This paper addressed the challenge of image clustering by integrating graph and orthogonality mechanisms into the nonnegative matrix factorization algorithm. It presented a novel model named graph regularized nonnegative matrix factorization with auxiliary variable orthogonal subspace (GNMFOSV). This innovative approach not only provided a rigorous proof of algorithm convergence using auxiliary variables, thereby filling a significant gap in the theoretical validation of similar algorithms under orthogonal conditions, but also effectively captured the nonlinear relationships in the reconstructed data. Additionally, it enhanced the sparsity of the decomposition results, leading to a notable improvement in clustering performance. To verify the effectiveness of the proposed method, comprehensive clustering tests were conducted on diverse datasets. The experimental results clearly demonstrated that the GNMFOSV algorithm outperformed existing methods in terms of clustering performance, indicating its great potential for practical applications.