Phosphorylation of tau at Ser (396, 404) (p-tau^396,404) is one of the earliest phosphorylation events, and plasma p-tau^396,404 level appears to be a potentially promising biomarker of Alzheimer’s disease (AD). The low abundance and easy degradation of p-tau in the plasma make the lateral flow assay (LFA) a suitable choice for point-of-care detection of plasma p-tau^396,404 levels. Herein, based on our screening of a pair of p-tau^396,404-specific antibodies, we developed a colorimetric and surface-enhanced Raman scattering (SERS) dual-readout LFA for the rapid, highly sensitive, and robust detection of plasma p-tau^396,404 levels. This LFA realized a detection limit of 60 pg/mL by the naked eye or 3.8 pg/mL by SERS without cross-reacting with other tau species. More importantly, LFA rapidly and accurately differentiated AD patients from healthy controls, suggesting that it has the potential for clinical point-of-care application in AD diagnosis. This dual-readout LFA has the advantages of simple operation, rapid, and ultra-sensitive detection, providing a new way for early AD diagnosis and intervention, especially in primary and community AD screening.