Layout Aware Textmaps: an approach to preserve two-dimensional textual structure of expense documents on machine learning based information extraction

Abstract

Automatic information extraction from scanned receipts and invoices is a task that is both widely researched and adopted in various businesses. With an ever-increasing demand on improving performance of business processes, the objective to extract accounting related information from expense documents was established at OP Kevytyrittäjä that offers invoicing, accounting and taxation service for light entrepreneurs. To preserve two-dimensional textual structure that is present in expense documents, a representation called textmap was formed. The textmap representation was used as an input for a fully convolutional network that was inspired by Inception network and U-Net. The produced model was a multi-output multi-class classifier that inferred price, expense type, VAT category and country of origin from receipts and invoices. For most variables an extreme imbalance existed between classes, and it had an extensive emphasis during the development process. The model achieved an accuracy of 89.5 % (price), 86.8 % (type), 90.2 % (VAT) and 98.9 % (origin) on imbalanced test dataset. Macro averaged F1 scores for type, VAT and origin, accounting for the data imbalance, were 72.4, 83.9 and 79.5 respectively. The textmap representation proves to be a feasible data format to work with two-dimensional textual structures. The quality of the data and the data size on minor classes were primary causes for the results not reaching exceptional levels. However, the model achieved a performance where it can be applied as a part of the production system. Further development has a great potential to reach outstanding results and scalability on new output variables.