AR headset names objects to guide blind people in new places

To assist the blind in navigating unfamiliar places, scientists at Caltech in the U.S. state of California have developed an augmented reality technology housed within a portable headset that gives objects a voice.

This week, Caltech announced the publication of a paper (“Augmented Reality Powers a Cognitive Assistant for the Blind”) outlining the work of scientists who “have combined augmented reality hardware and computer vision algorithms to develop software that enables objects to ‘talk.'” In layman’s terms, these researchers have constructed a headset powered by AR technology, called CARA, that says the name of objects in particular pitches and in particular locations to inform the wearer of what and where their surroundings are.

CARA, an AR assistant developed to help the blind navigate new places, was built for Microsoft’s HoloLens. Image: Microsoft

CARA stands for Cognitive Augmented Reality Assistant, which is the tech that is used within Microsoft’s HoloLens, a wearable headset computer that recognizes objects. The tech uses “spatialized sound,” or localized voices, to alert the user of where a particular object is — the closer an object is, the higher pitched “its” voice will be.

Of course, if the scientists stopped there, the headset will be saying tons of things to the wearer simultaneously which could no doubt be pretty alarming. Instead, the team programmed CARA with three modes to make understanding a space less overwhelming. Spotlight mode programs CARA to voice object names only where the user’s gaze is targeted at them. The second mode, scan mode, lists all objects in the environment that the headset can identify from left to right. Lastly, target mode allows the wearer to select a single object to “speak exclusively” with to use as a guide.

Although in its early stages, the technology is promising as all visually impaired individuals who tested the headset were able to successfully navigate a route guided by CARA on the first try. With algorithms for computer vision evolving and developing at such a rapid pace, the team looks forward to the possibility of this AR headset being used at banks, hotels, and malls, basically anywhere that navigating could be a challenge for the blind, sooner rather than later. NVG

RELATED STORIES:

Sharp vision: New glasses help the legally blind see

How the blind use clicks to ‘see’ – study