SoK: Benchmarking the Performance of a Quantum Computer

The quantum computer has been claimed to show more quantum advantage than the classical computer in solving some specific problems. Many companies and research institutes try to develop quantum computers with different physical implementations. Currently, most people only focus on the number of qubits in a quantum computer and consider it as a standard to evaluate the performance of the quantum computer intuitively. However, it is quite misleading in most times, especially for investors or governments. This is because the quantum computer works in a quite different way than classical computers. Thus, quantum benchmarking is of great importance. Currently, many quantum benchmarks are proposed from different aspects. In this paper, we review the existing performance benchmarking protocols, models, and metrics. We classify the benchmarking techniques into three categories: physical benchmarking, aggregative benchmarking, and application-level benchmarking. We also discuss the future trend for quantum computer’s benchmarking and propose setting up the QTOP100.