Systematic reviews (SRs), often conducted for evidence synthesis, require extensive multistage screening, with Stage‐Ⅰ title/abstract screening being especially time consuming. Covidence systematic review software, a widely used SR platform, has offered artificial intelligence (AI)‐assisted Stage‐Ⅰ screening function since 2022. However, this function's performance has not yet been formally compared across heterogeneous, large‐scale oncology‐focused SRs.

We selected two completed SRs conducted by the Program in Evidence‐based Care (PEBC), Ontario, Canada. The first SR (Axilla‐BC, N = 8774) supported a clinical practice guideline in breast cancer and the second SR (PET‐utility, N = 7267) informed the provincial Positron Emission Tomography (PET) steering committee in Ontario. Using each SR, we conducted a simulation‐based study by drawing subsets of 500, 1000, and 2000 articles and running 30, 30, and 10 simulation trials, respectively, to emulate Covidence's AI‐assisted Stage‐Ⅰ screening. For each trial, we calculated the workload and time savings at 95% and 100% sensitivity for relevant articles and at 100% sensitivity for finally included articles as primary outcomes. Secondary outcomes comprised missed finally included articles at 95% sensitivity for relevant articles. Wilcoxon rank‐sum tests were used to compare outcomes between these two SRs.

When pooling all subset sizes and the N = 1000 subsets, workload savings at 95% sensitivity were significantly higher for Axilla‐BC than for PET‐utility (median = 37.4% vs. 26.2%, W = 3156.5, p = 0.003 and median = 38.3% vs. 20.5%, W = 641.0, p = 0.005, respectively). Corresponding time savings were also significantly higher for Axilla‐BC (median = 6.0 vs. 4.2 h, W = 3089.5, p = 0.008 and median = 9.6 vs. 5.1 h, W = 639.0, p = 0.005). In Axilla‐BC, 20% of the 70 trials missed a total of 18 finally included articles, but only one PET‐utility trial missed one article.

In two large oncology‐SRs, Covidence's AI‐assisted Stage‐Ⅰ screening function demonstrated meaningful potential to reduce reviewer workload and screening time. Nonetheless, Covidence's performance varied notably between reviews, suggesting a possible association between the results and the characteristics of the SR, including topic, methodological complexity, relevance rate, and inclusion rate. Careful human oversight remains essential to ensure that no potentially finally included studies are missed in evidence synthesis, particularly in nuanced or methodologically complex reviews.

Conducting a systematic review (SR) is a time‐intensive process and represents the first phase in developing a clinical practice guideline (CPG). Completing a CPG through the Program in Evidence‐Based Care (PEBC), a globally acknowledged guideline program supported by Ontario Health (Cancer Care Ontario), typically takes about 2 years. Thus, expediting an SR can significantly reduce the overall time required to complete a CPG. Our recently published review identified two artificial intelligence (AI) tools, DistillerSR and EPPI‐Reviewer that reduced time in the title and abstract screening in an SR process when developing a CPG. However, the consistency and generalizability of these tools remain unclear within or across different SRs related to cancer. This study protocol aims to evaluate and compare the performance of DistillerSR and EPPI‐Reviewer against human reviewers for title and abstract screening (Stage Ⅰ screening) in cancer CPG development.

We will conduct a retrospective simulation study to evaluate and compare the performance of DistillerSR and EPPI‐Reviewer across 10 previously published CPGs by PEBC. These CPGs include the five cancer types with the highest incidence (lung, breast, prostate, colorectal, and bladder). We will run 30 simulation trials for one CPG per AI tool. Primary outcomes are workload savings and time savings in Stage Ⅰ screening. The secondary outcome is the percentage of missing articles among the final included articles. This informs the accuracy and comprehensiveness of the AI tools. Descriptive and inferential statistical analysis will be conducted to evaluate the outcomes.

This is a study protocol. The data presented in the tables are illustrative examples rather than actual study results, in accordance with the journal s standard structure. All data included in the final study will be thoroughly validated.

This will be the first study to investigate and compare the performance of DistillerSR and EPPI‐Reviewer in Stage Ⅰ screening of SRs in CPGs across different cancer types. These findings will inform the reliable use of AI tools in future cancer‐related CPGs. The results from this retrospective study will need to be confirmed by prospective studies.