Aggressive Speculative Decoding for Efficient Chain-of-Thought Reasoning

Large language models (LLMs) have demon-strated excellent performance in complex tasks through Chain-of-Thought (CoT) reasoning. However, LLMs usually gener-ate long thinking content during reasoning, leading to high inference latency. Speculative decoding (SD) is a general acceleration method but has two limitations that weaken its acceleration effect in CoT reasoning scenarios. Therefore, in this study, we proposed the aggressive speculative decoding (ASD) method for efficient CoT reasoning, alleviating the two shortcomings. First, SD adopts a strict acceptance strategy to ensure the generated results are consistent with the target model. However, multiple feasible reasoning trajectories may lead to correct answers. Therefore, ASD adopts an aggressive acceptance strategy, which accepts more draft tokens based on the deviation between the probabilities. Second, SD only generates a small fixed number of draft tokens in each drafting stage. In contrast, ASD adopts an aggressive drafting strat-egy, which adaptively generates a suitable number of draft tokens by estimating the expected transformation value of the inference latency through confidence. Extensive experiments on different reasoning datasets and LLMs demonstrate the effectiveness of our method. In particular, ASD achieved a significantly better acceleration effect than SD on all datasets while ensuring accuracy.