A Mobile Application for Nepali Sign Language Detection Using Deep Learning

Abstract

Sign language greatly benefits hearing-impaired individuals by enabling effective communication with the world. However, understanding and decoding sign language can be challenging for those unfamiliar with it. To bridge this gap, we propose developing a Nepali Sign Detection application using Flutter and leveraging ResNet-50 and VGG-16 deep learning algorithms. ResNet-50 outperforms VGG-16 with a higher F1 score (0.9333 vs. 0.8475), indicating superior precision, recall, and balance in detecting positive cases. To simulate actual network circumstances, we purposefully include delays in the creation of our mobile applications. It incorporates network inquiries using Flutter, camera access, and easier permissions management with ease. In order to mimic network latency, images are purposefully transferred to the Flask backend with pauses. To load, anticipate, and resize images, backend technologies like Pillow, Torch, and torchvision are utilized. To achieve accurate processing simulations, deliberate pauses are incorporated during model loading. The application's real-world robustness is increased by this purposeful delay integration, which guarantees authentic information flow. This application objective is to support both Nepali hearing impaired community and the people who are strange to Nepali Sign Language (NSL) in that they can recognize and understand NSL sign direct in real time.