Why should we care about spatial skills, i.e., the ability to visualize shapes in our “mind’s eye”? Here are a couple of relevant facts:

• Spatial intelligence is associated with higher performance in STEM subjects, especially math.
• Boys typically outperform girls on tests of mental rotation included in most IQ tests.
• There is evidence that early spatial ability predicts a young child’s reading skills.

Above is a classic rotation test, one measure of spatial intelligence. Are the two shapes different, or just oriented differently? If you felt smoke coming out of your ears trying to figure it out, all is not lost. People can enhance their spatial skills, sometimes dramatically, through experience and training with 3D objects.

• Physically manipulating 3D objects in structured block play, e.g., building a vehicle from Legos, is associated with increased spatial intelligence.
• Playing certain 3D 1st person shooter video games has been shown to improve spatial skills.

It was the last finding, with digital 3D characters and objects that are displayed on a 2D screen, that really piqued my interest. If we can improve spatial skills when using a 2D interface, how much more powerful might the effect be if the 3D objects are actually displayed in 3D, as in augmented and virtual reality?

We are just starting to see some research on this, so far coming from industry rather than academia… so let’s be a bit skeptical. Nonetheless, this blog post last year from Meta posits that providing IKEA furniture Instructions in 3D Augmented Reality results in faster assembly than the standard 2D print instructions.  As a cognitive psychologist, their reasoning for why this is true sounds plausible to me, i.e., cognitive overload, memory difficulties, perceptual inefficiencies.

Let’s take that last point and dig deeper. I think that the average Lego instructions are a marvel of design; their traditional 2D print approach has been honed over many years and millions of kids. Think about the advanced cognitive processes one goes through to successfully build a relatively simple Lego Friends project, pictured above.

1. Learn to follow step by step/page by page/number by number instructions, with each incremental step building on what came before.
2. Match the individual pieces pictured at the top of the page with similarly shaped and colored objects contained in the box. Find the correct number of each shape.
3. Place each object in the correct location; this is often a multi-step process and requires that you rotate the piece so that it lines up in the correct orientation.
4. If you reach a point where you can no longer carry out the instructions, you have to backtrack through the pages, deconstructing your model and learning where you deviated from the instructions.

How could Augmented Reality improve this process? The most obvious answer is to help pinpoint mistakes in real time, which otherwise can be a painful and lengthy process. Assuming your device knew what the model was supposed to look like at every stage, it could quickly pinpoint what piece was incorrectly placed using AR, showing it to you via a HUD-like interface. This process could also be used to double check your Lego creation at every stage, so you don’t end up having to disassemble it later.

More importantly, figuring out which piece goes where is the chief difficulty for most kids. This definitely requires the ability to look at a shape and rotate it (either physically or in your head or both) to see how and where it fits in the larger object. What if the child could view and rotate an AR version of the object as it appears at every stage, thus making it easier to see the correct placement? This would certainly improve “perceptual inefficiencies,” by reducing the need to convert the image from 2D to 3D and back again.

Spatial intelligence is related to math and science competency, not to mention directly tied to many different jobs in the workplace. Learning how to mentally rotate shapes to get to a particular solution is now recognized as a key underpinning of cognitive function. Whether we simply spend more time with our children building with blocks, or employ Augmented Reality to turn traditional 2D printed instructions into 3D visualizations, everyone will benefit.