How to Improve Memory for Virtual Items In AR/VR

Google’s Project Tango is the most interesting technology I have seen in a long while. It utilizes motion-sensing and depth-perceiving cameras to turn a mobile device into a “wayfinding” and Augmented Reality (AR) powerhouse! The special cameras on Tango devices enable “computer vision” that can map the area around you and detect your position in it.

The Project Tango dev kit unit.

The implications for interior mapping are huge. Using a phone with the requisite cameras, Google Maps will know where you are inside of any public building that has been scanned, and will direct you (perhaps with footprints visible through your phone) to the nearest bathroom, restaurant, elevator as easily as it now directs a driver to a gas station. What’s more, virtual items that appear around you are remembered by the device and even if you move on, when you return to that location the virtual item is still present and visible.

Capture

Some of the use cases for Tango technology from the Project Tango website.

Legacy is developing apps for kids on the Tango platform, and there are some interesting memory issues that arise almost immediately given these new capabilities. Imagine a child playing a digital “Where’s Waldo” game on this smart phone, during which they search their physical environment for hidden “treasures.” Should we assume that memory operates the same way in kids, regardless of whether or not the objects in their environment are real or unreal? Do children remember the number, placement and characteristics of virtual objects in a room, the same as they would for real objects? (Remember that virtual objects are only visible when seen through a screen.)

AR is relatively new in the tech world, which means very little specific research is available to guide our app design. Instead, we have to rely on some basic memory findings from cognitive psychology. Here’s a quick summary of the relevant research.

“Spatial memory” refers to memory for the position of things in the environment. Spatial memory can be long-term, e.g., remembering the route to your friend’s house, or it can use working memory, like when you know the position of cars around you as you change lanes on the freeway. “Working memory,” is a transient form of memory that holds information just long enough for you to act on it. (Try this surprisingly difficult spatial working memory test, here!)

Adults can typically hold 6-7 pieces of information in their working memory at one time. (There’s a reason why phone numbers are 7 digits.) For kids under 12, it’s typically fewer items. That means that if an AR game for children relies on spatial working memory, and assuming that it works the same way for virtual as well as physical items, there should only be about 4 or fewer objects for the child to actively keep in mind at one time. Otherwise, the user may experience mental overload and not be able to complete the task.

Here’s where it gets really interesting. Because Project Tango knows your location in a physical environment, we have some additional tools to assist the user’s working memory. Visual cues help us remember where things appear spatially. Do you ever think about surrounding landmarks when trying to remember how to get to your favorite restaurant? The best cues are distal cues, which are objects in the distance and have a fixed position (e.g. mountains, tall buildings). Proximal cues, i.e., objects right next to the item we want to remember, aren’t as helpful but are still better than nothing.

This is good news for Project Tango, because the surrounding environment is usually familiar (like the child’s room) and often already contains distal cues (e.g., pictures on the wall, windows, and doors). All the physical room cues available in AR will help children remember the positions of virtual objects that they can’t see without looking through a screen. (Compared to virtual reality, AR has an advantage in the spatial memory department because many visual cues come built-in to the real-world environment.)

toywheel-toy-car-rc-augmented-reality-app

An example of an AR app, Toy Car RC, that incorporates virtual objects in a familiar real-world environment.

And here’s the best news for our hypothetical hidden object game for Project Tango devices. Because Project Tango knows your location within a scanned room, the app designer can always spawn more objects close to where you are standing. Even if you forget where in the room you saw the treasure chest, the app designer can ensure that you will see it again regardless of where you are and how good or bad your spatial memory is. So ultimately, memory limitations do not have to limit the fun, assuming the designer uses all the capabilities of the technology.

Stay tuned for more explorations about designing for augmented reality and Project Tango. We’re just getting started!