Deep Learning-Based Instance Segmentation of Neural Progenitor Cell Nuclei in Fluorescence Microscopy Images

Abstract

In this work, a Deep Learning-based machine vision model was devel- oped for the detection, segmentation and counting of Neural Progenitor Cell nuclei

from fluorescence microscopy images. The cells were obtained from adult mice

and cultivated in vitro, with cellular nuclei labeled using DAPI dye. Convolu- tional neural networks for instance segmentation, specifically the Mask R-CNN

model with ResNet-50 and ResNet-101 backbones, were trained to recognize the

nuclei, and their results were evaluated. Nuclei labeling was implemented semi- automatically, applying a Superpixel technique and then refining the segmentations

from a manual process, also using a pre-trained model, which allowed to assem- ble a dataset of 66 images with 6392 labels in total. The results obtained with the

Resnet-50 backbone show that there is an effectiveness of 98.6% for between the specialist count and model-predicted count, in addition to having an mAP50 of 98.0%. This approach has the potential to significantly reduce the time and effort required to analyze large image sets, which is especially useful in studies that require repetitive and detailed cellular analysis.