As the primary carriers of population and economic activities, buildings are highly vulnerable to disasters, and their damage status directly affects emergency response and post-disaster reconstruction. Therefore, rapid and accurate acquisition of building damage information has become a critical requirement in disaster management. To address the challenges of geometric misalignment, background interference, and feature alignment difficulties in multi-temporal and cross-modal remote sensing imagery, we propose a Pre-Disaster Footprint-guided change-aware damage detection Network (PDF-Net). Specifically, the framework first employs a twin pyramid vision transformer to extract multi-level features from pre-and post-disaster imagery and generates pre-disaster building masks to introduce guidance information. Subsequently, a change-aware gated attention module is designed to enhance differential representation of low-level detail features, highlighting local changes, while a grouped cross-temporal attention mechanism with overlapped windows is introduced to explicitly align high-level semantic features, thereby reinforcing the structural change representation of buildings. Finally, fine-grained damage detection is achieved through cross-level feature fusion. Experiments conducted on the xBD dataset (pre-disaster optical-post-disaster optical) and the Bright dataset (pre-disaster optical-post-disaster SAR) demonstrate that the proposed method achieves significant improvements in both intra-modal and cross-modal tasks. Specifically, B-PriorNet surpasses the current state-of-the-art methods by 0.58% in mean Intersection-over-Union (mIoU) on the xBD dataset and by 1.97% on the Bright dataset, showing stronger robustness and generalization ability in cross-modal detection scenarios. These results validate the effectiveness and practical value of the proposed framework in complex disaster environments.