Segmenting collective lettuce to predict fresh weight using three-dimensional point cloud

Acquiring the individual parameters of collective lettuce under dense scenarios can greatly contribute to environmental regulation, yield prediction, and harvest timing determination in the growth monitoring center of the plant factory. Traditional monitoring can often involve the manual measurement of geometric parameters and root removal for fresh weight determination, leading to the less comprehensive and efficient. Fortunately, non-destructive monitoring can be expected to extract the crop phenotypic parameters using machine vision and machine learning at present. However, most existing machine learning exhibited certain limitations in the application of lettuce point clouds. For instance, the majority of application scenarios have been focused on the organ segmentation of individual crops. It is still lacking in the individual segmentation of collective plants. Additionally, the extraction of crop phenotypes from point clouds can often rely mainly on manually predefined feature quantities. There is a high demand to fully explore the effective phenotypic information within the lettuce point clouds. In this study, instance segmentation was proposed to process the point cloud data of collective crops. Subsequently, deep learning of point clouds was employed to predict the fresh weight of individual crops. The collective lettuce was also taken as the research object. A depth camera was also utilized to collect the single-plane point clouds of the collective lettuce. After point cloud preprocessing, the data was then input into the instance segmentation model (Mask3D) for training. A feature backbone network was employed to extract the features from the point clouds of the collective lettuce. A Transformer decoder was utilized to process the instance queries. Point cloud features were integrated with the instance queries through the mask module. A mask was then generated for each instance. The background and lettuce point clouds were segmented to distinguish the individual lettuce. Finally, the fresh weight prediction (FWP) network was employed to predict the fresh weight of individually segmented lettuce. The feature extraction network (PointNet) was utilized to extract the features from the segmented point clouds of individual lettuce. A multilayer perceptron was also employed to regressively predict the fresh weight of the lettuce. Experimental results indicated that the segmentation and extraction of individual lettuce point clouds were successfully achieved without over- or under-detection on the point cloud dataset. When the Intersection over Union (IoU) threshold was set to 0.75, the average precision (A_p) of instance segmentation was 0.924 for collective lettuce point clouds, superior to the instance segmentation models, such as Jsnet. Furthermore, the direct prediction of lettuce fresh weight reduced the errors associated with the manual feature extraction during processing using deep learning point cloud. The coefficient of determination (R²) and the root mean squared error (RMSE) were 0.90 and 12.42 g, respectively, indicating the superior accuracy of the fresh weight prediction network. Traditional and manual feature extraction from point cloud parameters with machine learning was achieved in the maximum R² of 0.83 and the minimum RMSE of 15.06 g. Therefore, the deep learning instance segmentation and point cloud regression can be expected to estimate the fresh weight of collective lettuce, indicating exceptional better performance. The finding can also provide significant importance for the growth monitoring, yield estimation, and harvest timing determination of facility vegetables.