Dynamic binary program analysis plays a crucial role in software vulnerability discovery and malicious code analysis. While 64-bit computing environments have become prevalent, numerous applications still utilize their 32-bit counterparts due to factors such as compatibility considerations, including both regular software and malware. Traditional dynamic analysis frameworks are not optimized for the analysis of 32-bit and mixed-mode programs. As a result, they often encounter issues such as anomalies and performance problems when applied to specific application analysis. To address these issues, this paper proposes WOWAF, an enhanced dynamic binary analysis framework tailored for Windows-on-Windows 64-bit (WOW64) environments that enables efficient fine-grained analysis of target applications. The framework is implemented on the built-in emulator in 64-bit Windows operating systems, facilitating effective and stable analysis of both pure 32-bit programs and mixed applications. By leveraging kernel features, the framework facilitates dynamic instrumentation of target programs, while incorporating a novel shadow memory allocation management scheme to minimize impact on program execution, and demonstrates its good deployment feasibility. The effectiveness of WOWAF is validated through comprehensive evaluations of diverse applications in real-world environments, such as exploit programs and evasive malware samples. Additionally, systematic benchmark experiments further demonstrate its strong analytical performance.

With the rapid development of operating systems, attacks on system vulnerabilities are increasing. Dynamic link library (DLL) hijacking is prevalent in installers on freeware platforms and is highly susceptible to exploitation by malware attackers. However, existing studies are based solely on the load paths of DLLs, ignoring the attributes of installers and invocation modes, resulting in low accuracy and weak generality of vulnerability detection. In this paper, we propose a novel model, AB-DHD, which is based on an attention mechanism and a bi-directional gated recurrent unit (BiGRU) neural network for DLL hijacking vulnerability discovery. While BiGRU is an enhancement of GRU and has been widely applied in sequence data processing, a double-layer BiGRU network is introduced to analyze the internal features of installers with DLL hijacking vulnerabilities. Additionally, an attention mechanism is incorporated to dynamically adjust feature weights, significantly enhancing the ability of our model to detect vulnerabilities in new installers. A comprehensive “List of Easily Hijacked DLLs” is developed to serve a reference for future studies. We construct an EXEFul dataset and a DLLVul dataset, using data from two publicly available authoritative vulnerability databases, Common Vulnerabilities & Exposures (CVE) and China National Vulnerability Database (CNVD), and mainstream installer distribution platforms. Experimental results show that our model outperforms popular automated tools like Rattler and DLLHSC, achieving an accuracy of 97.79% and a recall of 94.72%. Moreover, 17 previously unknown vulnerabilities have been identified, and corresponding vulnerability certifications have been assigned.