In this paper, we prove that all Coleman automorphisms of extension of an almost simple group by an abelian group or a simple group are inner. Using our methods we also show that the Coleman automorphisms of $2$-power order of an odd order group by an almost simple group are inner. In particular, these groups have the normalizer property.