Understanding how drivers perceive and respond to external stimuli in driving tasks is important for the development of advanced driving technologies and human-computer interaction. In this paper, we conducted a temporal response analysis between driving data and cortical activation data measured by functional near-infrared spectroscopy (fNIRS), based on a naturalistic driving experiment. Temporal response function analysis indicates that stimuli, which elicit significant responses of drivers include distance, acceleration, time headway, and the velocity of the preceding vehicle. For these stimuli, the time lags and response patterns were further discussed. The influencing factors on drivers’ perception were also studied based on various driver characteristics. These conclusions can provide guidance for the construction of car-following models, the safety assessment of drivers and the improvement of advanced driving technologies.