Why secret detection tools are not enough: It’s not just about false positives - An industrial case study

Checked-in secrets in version-controlled software projects pose security risks to software and services. Secret detection tools can identify the presence of secrets in the code, commit changesets, and project version control history. As these tools can generate false positives, developers are provided with mechanisms to bypass the warnings generated from these tools. Providing this override mechanism can result in developers sometimes exposing secrets in software repositories. The goal of this article is to aid software security practitioners in understanding why‘ secrets are checked into repositories, despite being warned by tools, through an industrial case study of analysis of usage data of a secret detection tool and a survey of developers who bypassed the tool alert. In this case study, we analyzed the usage data of a checked-in secret detection tool used widely by a software company and we surveyed developers who bypassed the warnings generated by the tool. From the case study, we found that, despite developers classified 50% of the warning as false positive, developers also bypassed the warning due to time constraints, working with non-shipping projects, technical challenges of eliminating secrets completely from the version control history, technical debts, and perceptions that check-ins are low risk. We advocate practitioners and researchers to investigate the findings of our study further to improve secret detection tools and related development practices. We also advocate that organizations should insert secondary checks, as is done by the company we studied, to capture occasions where developers incorrectly bypass secret detection tools.