This study focuses on microstructural and micromechanical modeling of abrasive sliding contacts of wear-resistant Hadfield steel. 3D finite element representation of the microstructure was employed with a crystal plasticity model including dislocation slip, deformation twinning, and their interactions. The results showed that deformation twinning interacting with dislocations had a key role in the surface hardening of the material, and it was also important for the early hardening process of the sub-surface grains beyond the heavily distorted surface grains. The effects of grain orientation and microstructural features were discussed and analyzed according to the micromechanical model to give a perspective to the anisotropy of the material and the feasibility of using micromechanics in virtual material design.