Picking Peaches or Squeezing Lemons: Selecting Crowdsourcing Workers for Reducing Cost of Redundancy

Crowdsourcing (CS) platforms are constantly gaining attention from both researchers and companies, due to the offered possibility of utilizing the “wisdom of crowds” in order to solve a great variety of problems. Despite the obvious advantages of such mechanisms, there are also numerous concerns regarding the quality assurance of work results produced by a large group of anonymous workers. In this work, we use data gathered from a real CS platform in order to study the performance of various approaches to worker selection, including a novel approach that utilizes automatic real-time monitoring of the produced results. We compare the performance of these mechanisms with respect to both overall cost and the accuracy of the final results to benchmark algorithms that aggregate results from a group of workers without pre-selection, relying solely on the “wisdom of crowd”. We find that our novel approach is capable of reducing the cost of obtaining high-quality aggregated results by a factor of four, without sacrificing quality.