Soybean (Glycine max L. Merr.) pubescence color is a trait commonly recorded by breeding programs. In previous research using high-throughput phenotyping (HTP), researchers could separate gray pubescence from light tawny and tawny pubescence, but could not separate light tawny from tawny. Using the Random Forest algorithm and time series of aerial RGB (red, green, blue) and multispectral images, this study aimed to classify pubescence color by testing models in data subsets from experiments grown over three years. By incorporating the pubescence color of the parental lines and training the model with a time series of images (four drone flights before or at maturity), a higher overall accuracy was achieved compared to a single flight at maturity. The red/blue index was the most successful feature for discriminating pubescence color, and the blue normalized difference vegetation index (NDVI) and green NDVI were also helpful, mainly in discriminating gray from light tawny pubescence. The overall accuracy was 86.55% in the best scenario (Kappa = 0.7976), and the sensitivity for gray, light tawny, and tawny pubescence were 0.893, 0.788, and 0.915, respectively. When models were tested in an independent environment, they achieved a lower overall accuracy of 65.86%, but still demonstrated fair to good model reliability (Kappa = 0.4874). Applying an HTP pipeline, as used in this study, would help breeding programs save time classifying pubescence color. Since pod color interferes with this trait in the background, genotyping a proportion of the plant rows for both traits and phenotyping pod color could improve the results.