Robotic systems require reliable tactile perception to evaluate object stiffness during physical interaction. This study proposes a lightweight dual-branch architecture, named Hybrid-CNN-ResVgg, designed to improve hardness recognition using data from a low-cost piezoresistive tactile sensor. The model combines a one-dimensional convolutional neural network (1D-CNN) based on a ResNet8-Lite architecture for learning temporal signal patterns and a two-dimensional convolutional neural network (2D-CNN) based on a VGG6-Lite architecture for learning spatial representations derived from Gramian Angular Difference Fields (GADF). A cross-architecture fusion mechanism is introduced to integrate temporal and spatial features while reducing redundant representation learning. Experiments were conducted on a controlled dataset comprising three hardness levels, with repeated grasp interactions to ensure consistent model evaluation. The proposed Hybrid-CNN-ResVgg achieved the highest accuracy of 89.67% among the evaluated models, including non-CNN baseline models, standard CNN architectures, tactile perception models, and single-domain lightweight CNN models. Despite its improved accuracy, the model requires only 0.039 giga floating-point operations (GFLOPs) and 0.46 megabytes of memory, supporting the computational feasibility of future real-time implementation on resource-constrained robotic platforms. The results indicate that combining temporal and spatial tactile information through lightweight cross-domain architectures can improve hardness recognition performance. This study provides a practical foundation for extending tactile perception toward more complex materials, continuous stiffness estimation, and multimodal sensing in future robotic applications.