Coverage-guided differential testing of TLS implementations based on syntax mutation

Transport layer security (TLS) protocol is the most widely used security protocol in modern network communications. However, protocol vulnerabilities caused by the design of the network protocol or its implementation by programmers emerge one after another. Meanwhile, various versions of TLS protocol implementations exhibit different behavioral characteristics. Researchers are attempting to find the differences in protocol implementations based on differential testing, which is conducive to discovering the vulnerabilities. This paper provides a solution to find the differences more efficiently by targeting the TLS protocol handshake process. The differences of different implementations during the fuzzing process, such as code coverage and response data, are taken to guide the mutation of test cases, and the seeds are mutated based on the TLS protocol syntax. In addition, the definition of duplicate discrepancies is theoretically explored to investigate the root cause of the discrepancies and to reduce the number of duplicate cases that are caused by the same reason. Besides, the necessary conditions for excluding duplicate cases are further analyzed to develop the deduplication strategy. The proposed method is developed based on open-source tools, i.e., NEZHA and TLS-diff. Three types of widely used TLS protocol implementations, i.e., OpenSSL, BoringSSL, and LibreSSL, are taken for experimental testing. The experimental results show that the proposed method can effectively improve the ability to find differences between different implementations. Under the same test scale or the same time, the amount of discrepancies increases by about 20% compared to TLS-diff, indicating the effectiveness of the deduplication strategy.