“The future is here, it’s just not evenly distributed yet.”
– William Gibson

The future is here and retailers are taking note.  We are in the midst of a major transformation in terms of how we shop, thanks to our ability to layer digital content on top of the real world. Combined with our increasing ability to turn past purchasing behavior into smart product recommendations, customers will soon be able to visualize in-store personalized promotions, coupons, and variable pricing alongside their favorite items on the shelf. The dual technologies – Augmented Reality (AR) and Artificial Intelligence (AI) – when paired together, will reinvent retail. 

When Augmented Reality is used in retail, the virtual and physical worlds are combined. Because the technology now exists to easily digitally map the interior of a store and know where every product is located, the customer can be directed step by step to the exact product they want.  Furthermore, when this location-based intelligence is combined with knowledge of the customer and their past purchasing behavior, we can ensure that their in-store experience is relevant to their preferences and more likely to lead to purchases.  

While there are many possible future scenarios regarding AR, AI and retail, my company, HitPoint Studios, has been researching which use cases resonate with customers and are technically feasible given current mobile devices. Retailers, however, are not all equally prepared to take advantage of what we learn about customer behaviors. From the get-go, companies like Amazon have a built-in advantage, compared to “legacy” retailers. Imagine how Amazon can and will transform the retail shopping experience, given their extensive knowledge of the specific customer and the ability of AR to display personalized information on top of the physical world.  

Amazon is already exploring what experiences and information the customer wants in their retail locations. Just in time for Thanksgiving, they announced that Amazon Prime members will be able to buy turkeys at a reduced price at Whole Foods stores ($1.99/lb. compared to 2.49/lb. for other customers). In the not too distant future, we will download an Amazon AR app and be able to see relevant special pricing and promotions, simply by pointing our phone camera over items in the freezer case or on shelves. Similar to how Amazon adjusts pricing on the fly online, the store turkey will cost more the closer it gets to Thanksgiving, and drop dramatically after turkey day. Furthermore, because of our online purchase patterns and the use of AI (Artificial Intelligence), we will be offered personalized discounts in the store. For example, I order avocado oil by the boatload on Amazon.com.  Amazon knows this and will offer me a virtual coupon as I walk past the oil shelf in Whole Foods.  Or, if avocado oil is out of stock on the shelf, I’ll be prompted to order it online and have it shipped to my home.  Similarly, customers will expect to access online rankings via Augmented Reality of every physical book sold in Amazon retail bookstores. And the transition between online and retail ordering, for digital or physical books, will be seamless.

Newer retailers, many of whom started out as online businesses, may have an easier time merging the online and physical world within their retail locations. Stores like Warby Parker and Sugarfina are not only “cooler,” according to millennials, but they know more about their customers. In addition, because their store locations are newer and often utilize the same inventory and planograms, mapping their physical interior and identifying the location of every item is a snap compared to older stores. Finally, I expect these formerly online-only stores to use in-store AR games and ties to social networks to attract younger buyers who want to share information and have a fun experience while shopping.

How do legacy retailers compete?  They are at a decided disadvantage, certainly. First, they need to know their customer.  Do they gather individual customer information, like purchase history and wish lists, that they can link to the customer who walks into their physical stores? How accurately do they track inventory and are products located in the same place in every store? Is there consistent pricing across stores? And more basic yet, do they have wifi throughout the store?  All of this is necessary information and infrastructure before AR can work its magic.

Someday your local retail store will know as much about you as Facebook, Google, and Amazon. Your online persona – your friends, preferences, purchases, and history – will be reflected in what you see and experience at your neighborhood mall. This is the promise and power of Augmented Reality and Artificial Intelligence, but its full implementation will probably arrive at some retailers sooner than others.

If you’d like to know more about the exciting AR projects we are working on at HitPoint, just drop me a line at [email protected].

Do you remember the Burger King web game, Subservient Chicken? A man in a chicken costume performed a wide range of actions based on the user’s input, showing pre-recorded footage. There were more than 300 commands that the “chicken” could follow. I remember watching a 7-year-old type, “Show me your butt,” and of course the subservient chicken did just that. It was hilarious.

This type of gameplay appeals enormously to kids because, at its core, controlling another character’s actions is highly empowering. Providing them with experiences in which they are the “boss” of another person, whether digital or real life, is immediately appealing. (I know many grownups for whom it works similarly.)

I believe that controlling increasingly life-like virtual characters will be at the heart of the most successful Augmented Reality (AR) experiences, and that classic play patterns will point the way towards satisfying gameplay and interactions.

With the advent of ARKit (Apple) and ARCore (Google), we have a new generation of 3D virtual characters to enjoy. Pet simulators are popular…surprise, surprise. Dress, feed, and play with Tamagotchi–like pets. The big difference is that these characters appear superimposed in your physical environment, as seen through your mobile phone. Keep up their “happy” meter, or they’ll run/fly away. These new AR virtual pets seem more life-like because they appear in the real world, and thus even more compelling to children.

You also control, albeit indirectly, virtual characters in HitPoint/Legacy’s new first-person game, Color BlastAR, on Apple’s ARKit. (A BIG update just launched, with extra-scary Halloween-themed graphics.) This fun walk-around AR game for kids of all ages combines tag and paintball with dragons, gnomes, yetis, zombies, and more for a very active experience!

Working on Color BlastAR has encouraged me to draw on other design inspirations, like classic outdoor multiplayer games, when thinking about virtual character interactions. What are some other interfaces, environments, and gameplay patterns that are relevant in designing an AR game?

1. Back to Subservient Chicken. What if we could directly control virtual characters, with simple commands (voice or text) like “turn around” or “touch your toes”? Could we re-create the game red light-green light with virtual characters? Could we control them with music, e.g., the character dances until we turn off the music, or as in musical chairs? Or perhaps we control them with our movement. As long as we continue to move our arms, the character will move.
2. Similarly, what if virtual characters could interact with each other, not just with the player? In Color BlastAR, I’d love to add a new gameplay mode – freeze tag. Once you color in a creature it remains frozen until another creature touches it, and unfreezes it! Meanwhile, you are still racing around trying to color all the zombies, orcs, dragons, ghosts, etc. before you get “chomped.” Would be frenetic but super fun!
3. I wish virtual characters were able to interact more with their physical environment. Unfortunately, occlusion doesn’t work well with ARKit and ARCore…yet. Characters walk through each other as well as through physical objects in their path. But even before occlusion is solved, the phone still knows something about the environment. We could use day/night, inside/outside, GPS data, etc. to make characters smarter about where they are and change up the gameplay accordingly. Maybe we make the ghosts in Color BlastAR only appear when you play outside at night?
4. Is it possible to combine target based and world-sensing AR so that virtual characters can interact with signs and symbols on the ground? I’d start with a virtual robot game, where it reacts to the symbols I draw in chalk (e.g., arrows) on the ground. If the robot steps off the correct path due to faulty programming logic, they disappear! (Naturally, you could control a virtual robot through programming from your phone as well.) Next we’ll be playing hopscotch with a zombie and designing mazes for our virtual friends!
5. What if virtual characters were invisible, while playing Hide and Seek or Marco Polo? You follow puzzle clues, both visual and auditory clues, until you get close enough to reveal the hidden character (who then pops out and scares the s— out of you!) Or perhaps you find the invisible character by throwing color or light around, and if it intersects with the character, that part of them is now visible. I’m smiling just thinking about it.

I haven’t discussed sports games with virtual characters, which could be massive, especially once AR Cloud and multiplayer is implemented in ARKit and ARCore. I’d love to let kids play Color BlastAR together, teaming up to paint the creature horde in tandem. (It’s on the road map.)

How important do you think interacting with virtual characters is to future AR gameplay?

I made a BIG career decision, to join HitPoint Studios. I have worked with this talented team in Massachusetts on four projects already, and am convinced that their tech and software development skills are incredibly well suited to pursue massive opportunities in Augmented Reality. Wish us luck!

I also want to say that I am incredibly grateful, at this stage of my career, to be able to do what I love best – envision and help build the very best games and experiences that we can create. We truly are at the beginning stages of what is possible, and I hope to contribute to reimagining learning and fun in a virtual world.

*A quote from Heraclitus

One week ago, Legacy’s first ARKit game, Color BlastAR launched. HitPoint Studios, Legacy’s exclusive development partner for the past two years, is publishing the game for us. What have we learned so far?

First, a brief description of Color BlastAR. A combination of coloring and tag, kids splat paint on life-size virtual zombies before they steal your color. There is a story game mode and also a fast-action arcade mode. The game utilizes the unique world-sensing, computer vision ARKit technology in order to integrate motion and SLAM tracking directly into the gameplay while transforming the world with interactive creatures, virtual flowers, rocks, treasure chests and more.

Here’s the good news. We were able to get some nice press coverage, in Venturebeat and CNET, among others. Kudos to our awesome PR consultant, Elizabeth Olson!

But here’s the bad news. We have not been featured by Apple and as a result, Color BlastAR has only been downloaded 10,000 times. Given that we were one of the highest rated and most downloaded games on Tango devices, this is very disappointing.  Clearly, an Apple feature is key to success at this early stage in the evolution of ARKit.

But we are indefatigable and have planned an awesome update on October 19. Here are some of the new features and content that is planned:

– Seasonal Halloween update including new icons, trailer, description and spooky Halloween environmental effects;
– “Paint My Dragon” story pack – a second storyline that adds 8 additional characters (IAP);
– Spooky Creature Pack, adds 10 new characters to the Arcade Mode including two brand new Halloween-themed creatures (IAP); and
– Dynamic shadows and maybe directional lighting to improve realism.

So stay tuned and, hopefully, we can turn 10,000 downloads into 100,000!

In my last blog, I wrote about Legacy’s design process and inspirations for creating a new cooperative mobile game for kids ages 4-7, based on the animated series, Kody Kapow, on the Sprout daily preschool programming block on Universal Kids. Our game, Kody Kapow Village Defender, is now available for free on iOS and Android.

How did the actual game development go and what did we learn?

Lesson #1. Assemble the Best Team. Legacy’s model, for many years, was to acquire a license and design the basic game concept, then hand the actual development process off to partners, usually far, far away. This worked great for well-established game genres, like match-3 and hidden object, and with standardized software libraries and engines. It didn’t work as well when we attempted to create an app for which there were few if any design precedents. The model really fell apart when we added brand new technologies like computer vision to the mix. It was hard if not impossible to ship yet-to-be released hardware out of the country as well as maintain the quick iterative dev cycles needed with constantly changing software and big time zone differences.

As a result, a few years ago when Legacy’s business became increasingly focused on AR and computer vision, we began to work exclusively with one of the largest independent game companies in Massachusetts, HitPoint Studios, headed by Paul Hake. The partnership has resulted in four games and counting. HitPoint’s combination of technical brilliance (they never say never), art versatility, and brilliant UI/UX has been the perfect addition to Legacy’s design and child development acumen. And possibly the most important member of the team, Andrew Duncan, functioned as the main game designer as well as producer of Kody Kapow. Virtually every good idea and magical moment in the game came from Andrew.

In addition to HitPoint and Legacy, the Digital Manager on the Universal Kids side couldn’t have been better, or easier, to work with.  Caroline Smigocki led the development effort for Sprout with humor, patience, and competence. We got every resource we needed, and she navigated approvals among the many stakeholders with amazing professionalism and speed.

Lesson #2. Iterate, iterate, iterate.

I’m still rather amazed at how much testing and subsequent revisions we did. Two of the minigames were straightforward and relied on well-known gameplay mechanics. Nonetheless, when we tested with kids in our age range, 4-7, we couldn’t find the right balance between engaging challenge and frustrating failure.

So what did we do?  Call the game doctor, that’s what! Mark Schlichting wrote the book, Understanding Kids, Play, and Interactive Design, which is the best book written on the subject, in my opinion. Mark agreed to consult on Kody’s overall game design, but it was his advice on the minigames specifically that turned out to be critical. He showed us how to break down the levels of difficulty into bite-size pieces, with careful scaffolding between levels. The children quickly learned how to play each minigame because of the onboarding and contextual help. As a result, there are no separate tutorials or endless voice-overs, despite the different types of gameplay introduced.

Lesson #3. Getting featured.

One would hope that creating a great game would be enough to get featured in Google Play and iTunes, but as any game developer can tell you, it is a necessary but not sufficient ingredient. We dutifully submitted our game to both stores, and went through the seemingly endless process of making the changes requested of us…this after we thought we were done! All of Google’s suggestions were reasonable and made the game work better on Android phones, e.g., insert the camera access request so it comes up only when needed, not at the beginning of the app. Check. The Android back button must have the same functionality as the back button in the game. Check. And lots of suggestions about how the pause button should function.

We believe that the feature opportunity with Apple, on the other hand, is tied to the iPhone 8 launch and ARKit. Given our previous experience with ARKit and Tango (3 apps and counting), we were able to quickly turn Mei’s mini-game into an ARKit experience. What fun, capturing 3D lanterns in your living room!

So will Kody Kapow Village Defender get featured by Apple or Google? The jury is still out, but it’s looking good!

Kody Kapow Village Defender is now live on Google Play and iTunes, for FREE. Try it out, with your favorite kid. Kody was a labor of love for us; we sure hope you enjoy it!

We love design challenges, but this was ridiculous. A children’s TV producer came to us about nine months ago and said, “I need a game that’s about teamwork and mindfulness, preferably both single player and multiplayer, for ages 4-7.” Sounds pretty daunting, doesn’t it? I couldn’t think of another cooperative digital game for kids that I really liked…plus the age range and COPPA laws further constrain the possibilities. Luckily, my colleagues at Legacy Games and I loved the TV show themes and characters and, together with our development partners at HitPoint Studios, embarked on a grand design challenge to create the game, Kody Kapow Village Defender.

Kody Kapow is a new original animated series for kids on Sprout, the daily programming block for pre-schoolers on Universal Kids. It features three adorable main characters – Kody, Mei, and Goji – as well as a dastardly villain, Mogo Monkey No-Go. Master Li, Kody’s grandfather, is on hand to teach the kids about how to use their superhero powers to help the villagers. There was a lot of great material to work with, including gorgeous 3D assets, witty voice overs, and engaging writing and storytelling, but what kind of gameplay would support the show’s heartwarming values?

DECISION #1. We first tried to tackle the design goal of creating a game that plays as well for one child as for three. In order to avoid complicating the game set up, and potentially raising COPPA compliance issues, we choose to focus on a pass and play, turn-taking style of multiplayer experience that would play the same regardless of the number of kids.  Check.

DECISION #2. Next, we wanted to tackle the problem of how to encourage kids to play together cooperatively, especially problematic if they are at different experience and maturity levels. We had to come up with a gameplay mechanic in which each’s child’s contributions could potentially help all the players win, or alternatively, all the players lose. Regardless, the players were in it together.

It turns out that physical board games provide some of the best examples of cooperative gameplay. (Cooperative games like Pandemic, Elder Sign: Omens, and Dead of Winter are popular and well-known to adult board gamers.) To our delight, we discovered Peaceable Kingdom’s popular cooperative board games, like Hoot Owl Hoot, for young children. Its basic premise is that game players must move all the baby owls along a colored coded track back to the nest BEFORE the sun rises (on a separate track). Game players work together to try and optimize their moves.

We borrowed the idea of two separate, color-coded game tracks, one for the protagonists in the story (Kody and friends) and one for the antagonist (Mogo Monkey No-Go). The game player has to figure out the optimum moves for each of the heroes to make, in order to get everyone to the Village before Mogo arrives. That involves problem-solving and strategic thinking, another one of our mandates from Universal Kids.

DECISION #3. We had a solid game concept, but now wanted to add some action elements and a light “retention loop” to keep the player engaged and coming back. Andrew Duncan, our Creative Director, designed fun mini games for each of the three main characters, uniquely suited to their personalities – one endless runner, one arcade action, and one Augmented Reality discovery game. The games become available when you choose to move a character to a specially marked tile. The more mini games the child plays successfully, the more lanterns they will collect. At the conclusion of the game, if the players beat Mogo to the Village, total lanterns are counted and high scores recorded.

Add to the mix a secret passage way and some funny antics from Mogo and his henchmen, and our design work was finished.

Or was it? Next up, how did the actual development proceed?  Stay tuned…

Kody Kapow Village Defender is available on Android and soon to be on iOS.

Have you ever played Phone Stack at dinner? Everyone places their smartphones in a pile in the middle of the table. The goal is to see who can last longer without reaching for their device. It is ridiculously hard, even though the loser usually ends up paying for the entire meal!

What if the mere visual presence of your phone reduces your cognitive abilities? In a recent study, “Brain Drain,” researchers have found evidence that phone addiction reduces our ability to perform mental tasks such as solving math problems and remembering letter sequences. What is surprising is that this finding occurs whether or not the phone is placed with the screen down, turned off and/or silenced. In other words, our thinking is negatively impacted by our device, whether or not we are directly interacting with it.

I believe it. We have limited cognitive capacity, and when we see our phone, scarce attentional resources are diverted from the task at hand. We are reminded about all the things we can do with our phone, messages waiting for us, our Facebook feed, and more. The only time the subjects in the study could effectively focus was when their phone was out of the room and out of sight.

I have one caveat. It’s worth noting that in this study, the effect sizes are small, barely significant. When you look more closely at the data, it seems to be driven largely by a small subset of people who are VERY ATTACHED to their phones.  Because of the important implications of this study, I hope others attempt to replicate the results and also compare to other forms of distraction, e.g., TV.

What are some implications of our constantly distracted state? Researchers have shown that when we are in this distracted state of mind, the increased cognitive load causes us to rely less on analytic and deliberative thinking, and more on intuitive, “emotional” approaches to decision-making. Advertisers and others who want us to buy something (whether products or ideology) already understand this and exploit it for their purposes.

So do yourself a favor this summer, go smell the roses, and ditch your phone for a while.

We all understand the joys of our always-wired world – the connections, the validations, the laughs…the info…But we are only beginning to get our minds around the costs.
-Andrew Sullivan (2016)

I just returned from a dreamy Mediterranean vacation, full of ancient Greek ruins, delicious food, and beautiful scenery. But as a newbie to archeology, appreciating 2500-year-old temples that look like rubble can be daunting. We used all the classic approaches of tourists – professional guides, self-guided audio tours, and read as much in-situ information as we could find. But by the time we reached the Valley of the Temples in Agrigento, Western Sicily, my sun-drenched brain realized that there had to be a better way to help visitors understand and appreciate the wonders of ancient Greece.

Augmented Reality, either mobile or wearable, offers a great way to educate tourists about ancient ruins and artifacts around the Mediterranean.  Here are five ideas of how AR could help future vacationers playing Indiana Jones in Greece and Sicily.

1. What visitor is going to read detailed informational signs like these? There is way too much text, and none of it personalized to my interests. While it’s nice that it is translated to English,  what about all the other visitors from around the world?

Alternatively, with AR, I could direct my own experience, by choosing where to focus my eyes or point my mobile device. I could choose to look at the top pediment of the temple, now sadly missing the “marbles” stolen by some random British sailor, and see how it used to look in all its glory.

2. It’s an even worse problem when there is no signage at all. Most of the time, I was clueless about what I was looking at. The Greeks built a wall that stretched for 15 km to defend Agrigento from the Carthaginians. (BTW, Mr. President, it didn’t work.) But I couldn’t understand why the wall had so many holes carved into it.  With my AR device, I could have pointed at one of the holes to find out that the wall did double duty as a gravesite.

3. Since I am not an art history scholar and have never been to Greece or Sicily before, I lack much of the background that makes an experience like this truly meaningful. I wanted to see what the ruins looked like when first built, without the missing columns and roof, and to scale. Did you know that ancient Greek buildings were beautifully painted and often decorated with precious metals and jewels? Many of the temples were erected in thriving cities of 100,000 people or more.  An AR app could show the buildings in their full glory and geographic context.

4. The ancient temples at Agrigento were spread out over a very long and hot three miles. Good luck with trying to find a bathroom or somewhere to buy water. We didn’t have a map with us because they ran out of English versions. AR mapping would have solved our navigation problems neatly.

5. It’s almost impossible to make a tour of Greek ruins and marble statues of gods interesting to kids. The addition of AR, however, allows for simple games that engage them in a meaningful and memorable way. For example, there was a wonderful garden at Agrigento that hosted many native, but unlabeled, fruit trees. How fun it would be to create a treasure hunt style game, where kids are rewarded for looking up into the correct tree branches with their AR camera and finding ripe fruit, regardless of the season. Do you know what fruit is pictured below? Pistachios!

One of the primary goals of art appreciation is to encourage the viewer to look more closely, and examine the art for patterns and similarities to other objects. Sicily hosts some of the finest Roman mosaics and frescoes in the world, but often there are large sections missing from a pictorial scene or abstract pattern. Why not show a section of the tile, and then ask the student to figure out where to insert it? Like a jigsaw puzzle, when all the pieces have been correctly placed using their AR camera or glasses, the famous blue monkey fresco, painted more than 3000 years ago, appears in all its original glory.

Not to pick on Greece and Sicily, because many of these same ideas can be applied to other tourist sites and museum exhibitions. Plus, think how useful an AR content management system (CMS) that is GPS aware would be to museum personnel. Experts could add and update the information themselves, controlling what text, VO, and images appear when the user points to or focuses at a particular location with their AR device.  Providing in-context and personalized information, that directly relates to where we are in physical space, is a potentially very powerful tool.

And so much fun!

I love to read anything and everything. When I can tear myself away from books like Richard Scarry’s Cars and Trucks and Things That Go (see above), I slog through piles of email newsletters, blog posts, books, podcasts. What follows are three things I’ve read recently that are perfect for developers of children’s apps, written by people I admire.

Shazia Makhdumi runs the family section of the Google Play store. She and her talented team get to see just about every kids’ app made and stay on top of all the latest trends. Given the current “gold rush” of developers submitting thousands of new AR/VR apps, Shazia is trying to figure out what quality standards are needed in this brave new world, as well as “best practices” for developers.

Shazia recently penned an article, “Inspiring Learning and Creativity with VR,” in which she described the unique features of Virtual Reality that make it particularly impactful for kids: presence, empathy, immersion, agency. Shazia goes on to provide helpful suggestions to developers, with examples, for building an impactful and positive VR experience for kids.

I believe that parents should proceed cautiously when exposing young children to Virtual Reality, making sure that gameplay periods are short. And I worry about kids and teens being exposed to hyper-realistic and immersive first person shooters…you know it’s going to happen once the hardware is in the home. Finally, the jury is out regarding the impact of current technology on the physiology of developing eyes.  It’s all so new. But one thing is inarguable. Kids love, love, love the technology. Parents will soon be faced with decisions about how much and which VR experiences they want for their children. I hope developers are up to the challenge as outlined by Shazia.

Warren Buckleitner is the man I hope most parents and teachers will turn to with their questions about new technologies. He is the editor of the best (in my humble opinion) review source for apps, Children’s Technology Review. If you make apps for kids, you need to subscribe to this service, now. The database of reviews goes back to 1993, and has recently become much easier to navigate and use; it’s the perfect resource for compiling those dreaded competitive product analyses.  Legacy’s games haven’t always received the review scores that I think they should, but even so, Warren is unfailingly warm and generous with his feedback and very consistent in the way he looks at products. (Hint: Immerse yourself in constructivism theory about how children learn and make your interactive designs open-ended whenever possible.)

In addition to running an app review service as well as one of the most prestigious conferences, Dust or Magic, Warren also teaches a college class in child development and app design. Called The Interactive Designer’s Cookbook, Warren and his students have summarized a variety of psychological theories and their relevance to app design, and posted the results online. It is a delightful read, especially rich on Piagetian and humanist influences.

My final reading recommendation is The Smarter Screen. Let’s get the advertisement out of the way first. The author of the book is the behavioral economist from UCLA, Shlomo Benartzi and my son, Jonah Lehrer, a science writer.

I love the book. It’s an easy read, chock full of great suggestions about how to be more effective when creating digital content. The basic premise is that we process information on screens faster, and not necessarily better. Screens make us more impulsive, reactive and careless, more prone to rely on first impressions. We tend to be driven more by emotions and instinct, rather than be reflective and analytical. As a result, we remember screen based content less well. There are ways around all of this, of course, with huge implications for designers.

What does this have to do with children? First, how do we slow down information processing on a screen, so that children are able to glean the content that we want to convey? Second, attention is everyone’s most valuable resource in a digital world.  Children have a more limited short-term memory and can easily become overwhelmed with too much screen content, suffering cognitive overload. How do we design clean screens with limited options, so it’s obvious where your eyes should go?  You’ll have to read The Smarter Screen to find out!

What are you reading these days and how does it influence your thinking about software design?

Why should we care about spatial skills, i.e., the ability to visualize shapes in our “mind’s eye”? Here are a couple of relevant facts:

• Spatial intelligence is associated with higher performance in STEM subjects, especially math.
• Boys typically outperform girls on tests of mental rotation included in most IQ tests.
• There is evidence that early spatial ability predicts a young child’s reading skills.

Above is a classic rotation test, one measure of spatial intelligence. Are the two shapes different, or just oriented differently? If you felt smoke coming out of your ears trying to figure it out, all is not lost. People can enhance their spatial skills, sometimes dramatically, through experience and training with 3D objects.

• Physically manipulating 3D objects in structured block play, e.g., building a vehicle from Legos, is associated with increased spatial intelligence.
• Playing certain 3D 1st person shooter video games has been shown to improve spatial skills.

It was the last finding, with digital 3D characters and objects that are displayed on a 2D screen, that really piqued my interest. If we can improve spatial skills when using a 2D interface, how much more powerful might the effect be if the 3D objects are actually displayed in 3D, as in augmented and virtual reality?

We are just starting to see some research on this, so far coming from industry rather than academia… so let’s be a bit skeptical. Nonetheless, this blog post last year from Meta posits that providing IKEA furniture Instructions in 3D Augmented Reality results in faster assembly than the standard 2D print instructions.  As a cognitive psychologist, their reasoning for why this is true sounds plausible to me, i.e., cognitive overload, memory difficulties, perceptual inefficiencies.

Let’s take that last point and dig deeper. I think that the average Lego instructions are a marvel of design; their traditional 2D print approach has been honed over many years and millions of kids. Think about the advanced cognitive processes one goes through to successfully build a relatively simple Lego Friends project, pictured above.

1. Learn to follow step by step/page by page/number by number instructions, with each incremental step building on what came before.
2. Match the individual pieces pictured at the top of the page with similarly shaped and colored objects contained in the box. Find the correct number of each shape.
3. Place each object in the correct location; this is often a multi-step process and requires that you rotate the piece so that it lines up in the correct orientation.
4. If you reach a point where you can no longer carry out the instructions, you have to backtrack through the pages, deconstructing your model and learning where you deviated from the instructions.

How could Augmented Reality improve this process? The most obvious answer is to help pinpoint mistakes in real time, which otherwise can be a painful and lengthy process. Assuming your device knew what the model was supposed to look like at every stage, it could quickly pinpoint what piece was incorrectly placed using AR, showing it to you via a HUD-like interface. This process could also be used to double check your Lego creation at every stage, so you don’t end up having to disassemble it later.

More importantly, figuring out which piece goes where is the chief difficulty for most kids. This definitely requires the ability to look at a shape and rotate it (either physically or in your head or both) to see how and where it fits in the larger object. What if the child could view and rotate an AR version of the object as it appears at every stage, thus making it easier to see the correct placement? This would certainly improve “perceptual inefficiencies,” by reducing the need to convert the image from 2D to 3D and back again.

Spatial intelligence is related to math and science competency, not to mention directly tied to many different jobs in the workplace. Learning how to mentally rotate shapes to get to a particular solution is now recognized as a key underpinning of cognitive function. Whether we simply spend more time with our children building with blocks, or employ Augmented Reality to turn traditional 2D printed instructions into 3D visualizations, everyone will benefit.