A Mobile Application for Keyword Search in Real-World Scenes

Keyword search in a cluttered environment is difficult in general, and even more challenging for people with low vision. While magnification can help in reading for low vision people, it does not facilitate efficient visual search due to the constriction of the field of view. The motivating observation for this study is that, in a large number of visual search tasks, people know what are they looking for (i.e., they know the keywords), they just do not know where to find them in the scene. We have developed a mobile application that allows the users to input keywords (by voice or by typing), uses an optical character recognition (OCR) engine to search for the provided keyword in the scene captured by the smartphone camera, and zooms in on the instances of the keyword detected in the captured images, to facilitate efficient information acquisition. In this paper we describe the development and evaluation of various aspects of the application, including comparing the various mainstream OCR engines that power the app, and an evaluation study comparing the app to the conventional optical magnifier vision aid. Normally sighted adults, while wearing blur glasses to lower their visual acuity, performed keyword searches for a series of items ranging from easy to difficult with the app and with a handheld magnifier. While there was no difference in the search times between the two methods for the easier tasks, the app was significantly faster than the magnifier for the difficult tasks.