Deep Feature Learning with Concatenated Rectified Pooling Units

This work introduces a multi-output activation function that combines the well-known single output Rectified Linear Unit (ReLU) with the operation of maximum pooling into a single four-output activation called Concatenated Rectified Pooling Unit (CRPU). We also introduce a separate variant of the CRPU activation with twelve outputs that is symmetric about the origin and call it the Symmetric Rectified Pooling Unit (SRPU). The activation functions are inspired by the need for building shallow neural networks for use on hardware limited platforms in Radio Frequency (RF) environments. We empirically establish the efficacy of these activations by implementing them on five classification datasets, four general image based datasets, and one RF dataset. Precisely, both activations are tested on the classical MNIST, CIFAR-10, CIFAR-100, and FashionMNIST datasets and a custom built RF Emitter identification dataset (RF-Emitter). The results of our experiments show that the CRPU and SRPU activations lead to faster convergence and similar or higher accuracy on all the datasets when compared to a set of baseline convolutional networks using four standard activation functions, three of them being single output and one being a multi-output, namely, ReLU, TanH, ELU, and CReLU. Moreover, the underlying networks do not overfit when compared to the corresponding baseline architectures and have lesser number of parameters in several configurations.