FBAM: A Frequency-Based Attention Mechanism for Enhanced Image-Based Malware Detection

The rapid growth and increasing sophistication of malware pose significant challenges to traditional detection methods. Convolutional neural network (CNN)-based malware image classification methods have emerged as a promising approach by transforming binary files into visual representations and enabling automated feature extraction. To enhance discriminative learning, recent studies have incorporated attention mechanisms originally developed for natural image and natural language processing tasks. However, these mechanisms embed inductive biases that assume spatial coherence and visually salient semantics, assumptions that do not necessarily hold in malware image representations, where informative patterns may be subtle, structurally encoded, and globally distributed. To address this representation–mechanism misalignment, this study proposes a Frequency-Based Attention Mechanism (FBAM), a domain-aware module that introduces a frequency-based feature transformation prior to attention computation. By converting feature maps into distribution-aware representations, FBAM enables spatial and channel attention to be guided by statistical feature distributions rather than raw activation magnitudes, allowing more effective capture of malware-specific patterns. FBAM was embedded into seven CNN architectures and evaluated on a dataset comprising 18,060 Windows Portable Executable (PE) files, including both malware and benign samples. Comprehensive experiments were conducted against general-purpose attention modules, including SE, CBAM, and CA, as well as the domain-specific SACNN model. Results show improved performance across accuracy, precision, recall, and F1-score, with statistical analysis indicating significant gains in most cases. VGG16 and VGG19 augmented with FBAM achieved accuracies of 98.82% and 98.38%, respectively, outperforming baseline architectures and competing attention mechanisms. These findings provide strong empirical evidence that incorporating distribution-aware frequency information into attention design enhances discriminative feature learning in image-based malware detection.