Detecting Anomalies in TLS Traffic Using Encrypted Traffic Analysis

Abstract

Subject of research was assigned by Cinia Oy. Cinia is a Finish company providing network, software and cyber security services and solutions. Subject was delineated to detection of anomalies in Transport Layer Security (TLS). Mitigation, response and forensics activities were out of scope. Primary objective was to gain understanding of methods and tools how one can detect anomalies in TLS encrypted traffic without decrypting it and how opensource products could be utilized. Another, more practical, objective was to investigate how two different commercial products, SensorFleet and LogPoint, could be combined and utilized as an Encrypted Traffic Analysis (ETA) solution. Opensource ETA solution, based on Security Onion and RITA systems, was used as a reference. Research was made using qualitative methods and analyzing was ongoing process during the whole research. First, theoretical data from books, research papers, web articles and videos were analyzed by researcher and then applied in testing phase. Results from tests was analyzed and followed by conclusions.