Contextualizing Environments with Semantic-Level Information: Scene recognition using computer vision and natural language processing algorithms

Abstract

Indoor scene recognition and semantic information can be useful for social robots. Recently, in the field of indoor scene recognition researchers have incorporated object-level information and shown improved performances in room classification. In line with these advances, this thesis aims to demonstrate that scene recognition can be performed solely using object-level information. YOLO, a state-of-the-art object detection algorithm, was trained to detect objects typically found in indoor environments and then used to detect objects in scene data. These predicted objects were then used as features to predict room categories. In this thesis, we successfully use object detection (YOLO) and term frequency-inverse document frequency (TF-IDF) to perform scene recognition by combining approaches used in computer vision and natural language processing. These approaches could be further useful in the field of embodied research and dynamic scene classification, which we elaborate on.