Strawberry picking point localization ripeness and weight estimation

Labour shortage, difficulties in labour management, the digitalization of fruit production pipeline to reduce the fruit production costs have made robotic systems for selective harvesting of strawberries an important industry and academic research. One of the important components of such technologies yet to be developed is fruit picking perception. For picking strawberries, a robot needs to infer the location of picking points from the images of strawberries. Moreover, the size and weight of strawberries to be picked can help the robot to place the picked strawberries in proper punnets directly to be delivered to customers in supermarkets. This can save significant time and packing costs in packhouses. Geometry-based approaches are the most common approach to determine the picking point but they suffer from inaccuracies due to noise, occlusion, and varying shape and orientation of the berries. In contrast, we present two novel datasets of strawberries annotated with picking points, key-points (such as the shoulder points, the contact point between the calyx and flesh, and the point on the flesh farthest from the calyx), and the weight and size of the berries. We performed experiments with Detectron-2, which is an extended version of Mask-RCNN with key-points detection capability. The results show that the key-points detection approach works well for picking and grasping point localization. The second dataset also presents the dimensions and weight of strawberries. Our novel baseline model for weight estimation outperforms many state-of-the-art deep networks. The datasets and annotations are available at https://github.com/imanlab/strawberry-pp-w-r-dataset.