Introducing attention shortcuts in convolutional neural networks

It is a proven fact that nowadays, thanks to convolutional neural networks that implement skip connection mechanisms, we can train increasingly deeper, more accurate, and efficient networks. These networks successfully address the degradation problem previously experienced in very deep networks. Among the most popular are ResNets and DenseNets. ResNets introduce skip connections using summation, while DenseNets employ concatenation. The summation mechanism in ResNets can limit the adaptation of prior information to the specific needs of each layer. In contrast, the DenseNet concatenation mechanism can become computationally expensive as convolutional blocks attempt to process all accumulated prior information. Therefore, in this paper, we proposed a new attention-based skip connection mechanism: Attention Shortcuts. This mechanism allows convolutional blocks to process the most relevant prior information, reducing computational burden. We conducted a preliminary experimental study adapting the proposed mechanism to the ResNet-50 backbone and compared its performance to the original ResNet.