Will AR Help Bridge the Gap in Spatial Intelligence?

Why should we care about spatial skills, i.e., the ability to visualize shapes in our “mind’s eye”? Here are a couple of relevant facts:

  • Spatial intelligence is associated with higher performance in STEM subjects, especially math.
  • Boys typically outperform girls on tests of mental rotation included in most IQ tests.
  • There is evidence that early spatial ability predicts a young child’s reading skills.

Above is a classic rotation test, one measure of spatial intelligence. Are the two shapes different, or just oriented differently? If you felt smoke coming out of your ears trying to figure it out, all is not lost. People can enhance their spatial skills, sometimes dramatically, through experience and training with 3D objects.

  • Physically manipulating 3D objects in structured block play, e.g., building a vehicle from Legos, is associated with increased spatial intelligence.
  • Playing certain 3D 1st person shooter video games has been shown to improve spatial skills.

It was the last finding, with digital 3D characters and objects that are displayed on a 2D screen, that really piqued my interest. If we can improve spatial skills when using a 2D interface, how much more powerful might the effect be if the 3D objects are actually displayed in 3D, as in augmented and virtual reality?

We are just starting to see some research on this, so far coming from industry rather than academia… so let’s be a bit skeptical. Nonetheless, this blog post last year from Meta posits that providing IKEA furniture Instructions in 3D Augmented Reality results in faster assembly than the standard 2D print instructions.  As a cognitive psychologist, their reasoning for why this is true sounds plausible to me, i.e., cognitive overload, memory difficulties, perceptual inefficiencies.

Let’s take that last point and dig deeper. I think that the average Lego instructions are a marvel of design; their traditional 2D print approach has been honed over many years and millions of kids. Think about the advanced cognitive processes one goes through to successfully build a relatively simple Lego Friends project, pictured above.

  1. Learn to follow step by step/page by page/number by number instructions, with each incremental step building on what came before.
  2. Match the individual pieces pictured at the top of the page with similarly shaped and colored objects contained in the box. Find the correct number of each shape.
  3. Place each object in the correct location; this is often a multi-step process and requires that you rotate the piece so that it lines up in the correct orientation.
  4. If you reach a point where you can no longer carry out the instructions, you have to backtrack through the pages, deconstructing your model and learning where you deviated from the instructions.

How could Augmented Reality improve this process? The most obvious answer is to help pinpoint mistakes in real time, which otherwise can be a painful and lengthy process. Assuming your device knew what the model was supposed to look like at every stage, it could quickly pinpoint what piece was incorrectly placed using AR, showing it to you via a HUD-like interface. This process could also be used to double check your Lego creation at every stage, so you don’t end up having to disassemble it later.

More importantly, figuring out which piece goes where is the chief difficulty for most kids. This definitely requires the ability to look at a shape and rotate it (either physically or in your head or both) to see how and where it fits in the larger object. What if the child could view and rotate an AR version of the object as it appears at every stage, thus making it easier to see the correct placement? This would certainly improve “perceptual inefficiencies,” by reducing the need to convert the image from 2D to 3D and back again.

Spatial intelligence is related to math and science competency, not to mention directly tied to many different jobs in the workplace. Learning how to mentally rotate shapes to get to a particular solution is now recognized as a key underpinning of cognitive function. Whether we simply spend more time with our children building with blocks, or employ Augmented Reality to turn traditional 2D printed instructions into 3D visualizations, everyone will benefit.

A New Legacy

Have you visited LegacyGames.com lately? You will notice many changes! Gone are our 200+ downloadable games, retail bundles, Facebook integration, etc.

While bittersweet to take down all of the PC games that Legacy has created over the past five years, including my personal favorites Murder, She Wrote and Criminal Minds, it was time. It’s not a secret that downloadable hidden object games are no longer in vogue. What was once a booming business is no longer viable. Trends come and go rather quickly in the video game industry, and you either “pivot” or die.

So…we embarked on a rebranding of Legacy a few months ago, to more accurately represent our current business focus.

The “new and improved” Legacy Games website reflects our passion for invention, human-centered design, and product solutions that both exploit and promote new technologies. Our Crayola Color Blaster augmented reality game has been a magical project for us, one that’s opened many doors in terms of new business opportunities. Building on that experience, we are now busily creating new AR/Tango apps as well as an audio-only game utilizing Amazon’s Echo.

And while we no longer offer PC games on our website, you can still find retail boxes with Legacy’s hidden object games at Walmart stores. See below for the latest disc compilations hitting retail shelves this week.

So not everything is new! Check out Legacy Games if you get a chance, and let me know what you think!

More Virtual Creatures to Chase and Be Chased!

 Today, Legacy is announcing an updated version of its unique augmented reality game, Crayola Color Blaster. A new story and content pack, complete with dragons, fairies, ogres, hydras, gnomes, will be joining the original zombie color crunchers. And by popular demand, an arcade mode has been added that requires less walking around, thus making it easier to play in a small space.

– Run from cartoony creatures as they follow you, then paint them with your Crayola Color Blaster!

– Get up close and personal with life-sized zombies, dragons, fairies, ogres, hydras, gnomes, and more!

– Defeat the wacky, rhyming Professor before he steals all the color from the world!

– Face endless waves of fantastic creatures in Arcade Mode and beat the high score!

– Tango’s world-sensing technology scans your environment and transforms it with virtual flowers, rocks, treasure chests, and more!

Enjoy!

The Definitive Podcast…about me

I love talking about myself but honestly, this recent interview with friend and fellow developer, Chris Natsuume,  is so long that I wasn’t even sure my family would listen to it. (My part starts about 1/3 of the way into the podcast.)

However, if you have ANY questions about the numerous zigs and zags it takes to run a small business, you may find it interesting.  There is a lot of ground to cover. I’ve founded three small software companies and developed product on technology platforms ranging from Apple II and PC MSDOS to Android-Tango and Amazon Echo.

A couple of highlights from the podcast:

  • I went to graduate school when Marshall McLuhan was all the rage. My dissertation, about how we process information differently depending on the medium, is still very relevant. Turns out the results of that study, especially how we understand and remember audio-only content, is informing our design of stories and games for voice assistants like Alexa and Siri.
  • Naturally Chris wanted to know about my biggest failure. That’s easy. It’s was during the Great Recession of 2009. I was caught completely flatfooted by the free falling economy. The bank called in the company credit line, and it was only with a large personal cash infusion (basically our retirement savings), that we were able to save the company.  That is not something I recommend doing, by the way, but luckily it worked out…by the skin of our teeth!
  • And everyone always wants to know what it was like to work with Google on their new Augmented Reality Tango platform. I talk about the process and resulting product, Crayola Color Blaster. (We are launching a new version of the game next week!)
  • Finally, Chris is a dad and asked about what it was like to raise kids 30 years ago compared to today and how the ubiquity of screens impact their relationship to technology. Good discussion about kids and new technology.

So hopefully there might be a few nuggets in there, if you have the time to listen. Thanks, Chris, for your generous spirit and lovely comments. Please enjoy!

P.S. My son, Eli, whom I mention in the podcast…the one who watched too much TV and became a TV executive? He actually did listen to the podcast and had one comment…”I’m not 40 yet!” I’m correcting the record here. He’s actually 39.

 

The Evolution of Play in a Virtual World

I am speaking about “The Evolution of Play in a Virtual World” on Tuesday, at Kidscreen. I will be comparing play patterns using traditional toys and games, mobile games and AR/VR games. My basic premise is that AR/VR has some important advantages over previous generations of digital games and also includes some of the unique features that make traditional games so compelling.

Not to give away my entire talk, but here are some of the benefits of Virtual Play that I will be discussing:

  1. You can physically interact with realistic 3D characters, chasing them, etc.
  2. In Augmented Reality, the real world environment becomes part of the game.
  3. The types of interfaces available for interacting with the game tend to be direct and impactful, including touch, voice, and your whole body.
  4. It promotes a sense of presence as you move beyond 2D screens to 3D virtual worlds.
  5. Social interaction and sharing become possible in a realistic 3D space.
  6. Physical movement is an essential part of the game…no more couch potatoes!

Of course, so much has to do with how well a toy or game is designed, regardless of whether it is digital or physical in nature. Developmental psychologists value open-ended children’s products, so that the way they are used is limited only by the child’s imagination.  They value toys and games that engage a child’s interest and curiosity, without distracting them with an over abundance of bells and whistles. And they value products that have repeat playability, with sufficient variability and complexity to explore over time.

Girl playing with Playdough

My mantra as a parent was always the word “balance.” I wanted our four children to experience content on every available platform, because I felt that learning how to “decode” the medium was an important life skill.  It made no sense to me to take away TV or computer time; I simply limited it to 60 minutes a day. The key was finding content that enriched their minds, engaged their hearts, and promoted the values that were important to me.  Those kinds of kids’ toys and games you can find in every format, including new technologies.

The Top CES Technology for Kids

CES 2017 badge

CES is not the ideal place to showcase kids’ products. One hall is filled with car technology, another with smart health care devices, another with TVs, phones, etc. Even porn made a comeback this year, courtesy of virtual reality!

But if you’re persistent, you might find gold, as well as some products that are just so silly you can’t believe they made it to the convention floor. The one technology in particular that I think over time will have a big impact on kids’ products is artificial intelligence (AI) via voice activated assistants.

I wrote in a previous blog about our family’s love affair with Google Home and Amazon Echo. Not surprisingly, there are a couple of big companies trying to productize this technology for kids. It’s challenging, as you might expect. Children’s speech patterns are difficult for most smart speakers to decipher, and they make even more mistakes than usual trying to interpret their voices.  Have a 5-year-old ask Alexa, “Who is the 25th president?” and see what happens. It’s not pretty.

Despite this, I believe that voice assistants will become the primary interface for many children’s digital products. Every designer of kids’ products understands the difficulty of presenting choices to children, especially when text/reading is not a good option and you’re faced with a tiny screen. And having a device that is responsive to a child’s every question…all 200+ of them daily…seems promising to me.

Stepping into this minefield is Mattel, a company with history in the space and the gumption to take on the challenge. Mattel’s previous voice activated foray was with the Hello Barbie doll, followed by her Dreamhouse, both of which sparked almost instant controversy. My concerns lay less with privacy issues and more with the fact that my grandchildren, Jude (7) and Teddy (5) found the pre-scripted interactions with the doll frustrating and unsatisfying. It seemed that no matter what we wanted to talk about, Barbie would respond with an unrelated topic, e.g., careers or horses.

This summer Mattel is launching a speaker-based voice assistant called Aristotle, that functions like Google Home or Amazon Echo, but is designed for kids and parents. Unlike the Company’s previous products, Aristotle relies on the AI driven by Amazon’s Alexa and Microsoft’s Cortana. In addition, the device functionality changes as the child grows. Initially it can be a smart baby monitor, and respond appropriately to a crying baby with soothing sounds, as well as order diapers on Amazon. With toddlers, it will evolve into a teaching tool for colors, numbers, and the alphabet. For older kids, it will help with homework, teach a foreign language, and even introduce two-player voice games.

Aristotle by Mattel/Nabi

In addition to the smart speaker, Aristotle also includes a camera, which opens up lots of possible game and toy applications. Imagine using it with a special mat and Hot Wheels, so that any time the camera sees the fire engine leaves the fire station, you hear the siren. Or Melody, an American Girl doll, could be recognized by the camera and given a voice and personality through the speaker that reflects her 1960’s world. A child’s favorite book, e.g., The Hungry Caterpillar, could be read out loud and on command. Mattel plans to open up Aristotle to 3rd party developers, which will ensure a rich pipeline of applications.

Of course, all of this relies on Aristotle actually understanding what is being asked, a definite problem with kids. Mattel’s solution is to introduce voice training, through which the device learns the nuances of the child’s voice. That’s the theory; when the device ships this summer we’ll see how well it actually works. Another potential issue with these types of devices for kids, as mentioned previously, are privacy concerns. Aristotle, like Echo and Google Home, are on all the time, listening for a specific word, or cry. The information is captured on Aristotle’s servers, and forwarded to the parent’s phone. What happens if/when this information gets hacked? And how do we protect kids from the dark side of the internet and sites that we don’t want them to access?

Turns out most parents are not very concerned…yet.  According to this study, parents like the convenience of sensors and all different types of connected devices that make up the Internet of Things, e.g., baby health tracking devices, smart baby/children’s toys, smart home/home security devices, smart nursery/baby gear, and of course, smart speakers with virtual assistants. A third of the parents questioned even believe that these devices make them a better parent!

 From Fisher-Price's "The Future of Parenting" video

From Fisher-Price’s “The Future of Parenting” video

While I wouldn’t go that far, I am enthusiastic about the idea of kids having access to the same amazing wealth of information on the internet, without needing a smart phone.  And the curiosity of every child is something that should be carefully nurtured. Whether or not an artificially intelligent character who listens and responds to questions will provide that encouragement, only time will tell.

My Highly Idiosyncratic Picks for Holiday Gifts

It’s already late in the season for a list like this…even writing and reading this blog is a form of procrastination! But I have a few favorite gift tips that I wanted to share.  So without further ado.

For competitive kids who like to play games, but not together
How do you get a 7 year-old boy to play with two 5 year-old girls, and not cream them every time at a game? My granddaughters, Rosie and Teddy, were tired of losing, and refused to play with their cousin Jude anymore. Naturally, I couldn’t allow the girls to quit so I found, while searching on Amazon, a company I love, Peaceable Kingdom, and their line of cooperative board games. Starting at around 3-4 years of age, I recommend the game, Hoot Owl Hoot, and progressing to other games like Race to the Treasure for slightly older kids. Everyone wins or everyone loses. For the first time, I watched the children help each other figure out the best moves. Very gratifying for this grandmother!

For the business partner who likes a good story
I read about this book first on Bill Gates’ blog. I won’t repeat all the amazing reviews, just buy it. The fascinating, funny, inspiring “Shoe Dog: A Memoir by the Founder of Nike” by Phil Knight, will be appreciated.

For anyone who designs kids games
Mark Schlichting has been an admired designer since his earliest days at Broderbund. In addition to having credits on many classic kids games, he just wrote what I consider to be the best book available on the art and craft of designing children’s interactive products.  It is chock full of wisdom and warmth throughout the almost 400 beautifully illustrated pages. My favorite chapters include a brilliant discussion of traditional children’s play patterns, and how the best designers tap into those patterns in order to create something new. Other helpful sections discuss how to design challenges, leveling and pacing, all of which is extremely difficult to get right in kids products. But the best part of the book, in my opinion, is Mark’s discussion of gendered play patterns. It is a minefield, without a doubt, but he manages to put into words so many of the sex differences I have observed in children over the years, and how that relates to design. Friendship versus winning, action versus dialogue, epic story versus personal relevance, and more. Don’t miss it: Understanding Kids, Play, and Interactive Design.

For the man who has everything
My husband, David, has a new love in his life, “OK Google”. He talks to his new Google Home device each morning in dulcet tones, saying “please” and “thank you” as the voice activated know-it-all tells him the traffic, plays his favorite playlist and much more. At first Google Home retrieved his requested tunes on Google music, but after only a few times of specifying Spotify, Google grocked David’s preference. Machine learning is a beautiful thing and is probably why we will end up using Google Home more than our Amazon Echo, another voice activated smart assistant device.

If you want to be the coolest parent, or grandparent
Forbes just published this list of the best connected toys this Xmas. Connected toys combine the best of physical toys with added digital pizzazz and depth. I have always been a fan of connected toys because kids learn through physical manipulation — not everything can or should be digital. It’s the combination of physical with digital where the real magic happens. Of the products listed, I am a big fan of the Osmo products (although ridiculously pricey) and Maze by Seedling. Kids, ages 8+, build their maze with physical pieces, take a picture of it, then put on Google cardboard and walk through their maze in virtual reality. They can further customize and decorate their maze with obstacles, pictures, etc. Very cool, and although I am not a huge fan of VR for kids, short play periods with Google cardboard feels like a good introduction to the experience that all kids seem to crave.

If you need to butter-up a teacher
I recommend another connected toy that just started shipping this week, called Happy Atoms. You physically assemble models of molecules from atoms, which connect via magnets.  Then you scan the molecules you’ve built, and the app provides detailed information about its composition, name, properties, etc. Designed for ages 10-17, it combines the best of physical manipulation, discovery learning, and digital depth.

If you still aren’t convinced, maybe follow the advice of this article. Research tells us that the best gifts are the ones that people actually ask for!  Shocking, I know.

Happy Holidays!

Is Tango the Future of AR?

The much anticipated first Tango-enabled device is finally released to the public, along with Legacy’s new Augmented Reality game, Crayola Color Blaster. Where do we go from here?

promo_2For Legacy, the answer is easy. We have new content for Color Blaster releasing Valentine’s Day 2017. Instead of running away from zombies, you’ll have to avoid rampaging gnomes, orcs, and dragons. Seriously, we will be focusing most of our future efforts on evolving the types of interactivity possible with virtual characters, given our discovery of how much fun this is in AR.

unnamedWhat about other Tango apps? Phantogeist is another great game with roughly humanoid shaped aliens invading our world, along with monstrous creatures of different sizes! Both Woorld and Raise bring cute little characters and objects into your room. As the Tango technology becomes better and smarter, virtual characters will be able to recognize what objects are. Imagine virtual humanoids relaxing in your favorite armchair, digital pets sleeping in your bed, a 3D chef inspecting your kitchen and pots, and more!

hotwheelsAnother thing Tango devices do well is the ability to track your movement in space. This helps construction games in particular. Placing straight tracks, curves, accelerators, and much more is intuitive in the Hot Wheels Track Builder because you can pick up the objects and walk it over to exactly where you need it. The Towers for Tango game lets you build skyscrapers on your coffee table. Move your device up and down each floor and get an X-ray view to customize things exactly the way you want.

So back to the original question…Are Tango-enabled smart phones the future of AR? Or is looking through a phone as you walk through a room or building just too wonky to become mass market? Pokemon Go would indicate otherwise. There will certainly be more Tango smart phones to hit the market in the next year, including some announced at CES in January. (Just google it!)

On the other hand, even if Tango-enabled smart phones don’t explode in popularity, it doesn’t take insider information (of which I have none) to predict that Tango software may become embedded in Google Daydream VR headsets. Once that happens, Daydream will become a mixed reality platform (it currently just supports VR), and its form factor will just keep getting smaller and more comfortable.

My point is that I believe Tango software and 3D camera technology is here to stay. Its 3D mapping usefulness, fun factor, and relatively low cost will make it ubiquitous, eventually. The form it takes, however, will vary, at least in the short term. Mobile phones now…headsets next…glasses later? What do you think?

screen-shot-2016-09-24-at-09-39-06-796x364

6 Research Questions for AR/VR

We know surprisingly little about the long term effects of Augmented and Virtual Reality, on both kids and adults. The technology, thanks to large investments of resources and talent, is proceeding faster than our understanding of how to harness it or its consequences. Software developers and hardware manufacturers desperately need the insights obtainable only through verifiable research, before establishing best practices and policies.

chicago-toy-fair

Don’t get me wrong. I am an unabashed technology enthusiast. I believe that AR/VR offer the most exciting new business opportunity since the invention of smart phones. Legacy just completed our first major AR project, built for new “world-sensing” Tango devices. Yet producing Crayola Color Blaster on this remarkable new mobile platform has raised many cognitive and developmental questions for our team. Unfortunately, there is a dearth of reliable, valid research to guide our many design questions.  For example:

  1. How does a VR experience affect, if at all, the child’s developing brain and vision? What is the long term neurological impact of the two small LCD monitors and lenses, each projected at one eye, creating a stereoscopic effect and giving users the illusion of depth? Some hardware manufacturers have established age recommendations, but we already know that children are intensely interested in VR and will be using the devices regardless.making-immersive-virtual-reality-possible-in-mobile-27-638createstereoscopicimages
  2. In a world with unlimited “virtual garbage” potential, how do we direct the user’s attention towards relevant data and away from irrelevant stimuli? What does cognitive load research tell us about how to manage limited attentional resources? This may be our toughest design challenge. 571534199
  3. What are the societal implications of a world that has been customized to reflect our interests? What if our experience of the world is personalized through AR glasses, much as our Facebook feed is currently? Will this just further our descent into silos of interests and beliefs?
  4. How can we create more authentic digital tests of student or employee ability using AR/VR? What if, instead of problem solving tasks delivered via a computer or tablet, we embed virtual tasks in the real world instead? Can we build assessment tools that truly capture someone’s range of talents?
  5. Does memory work the same way for virtual objects as it does for real ones? In our AR game, Crayola Color Blaster, children seemed to have a hard time remembering the placement of virtual items in the room. If memory for virtual objects is worse than for physical objects, we will have to add more spatial cues to AR applications and design around that deficit.
  6. Can we operationalize a “sense of presence,” which many people believe results from an immersive VR experience? Are there corollary physiological measures? Does it lead to increased empathy, as some researchers maintain? Does this also imply that violent, FPS VR apps will have an outsized impact on our emotions?

I was a graduate student 30 years ago in Cognitive Psychology. Almost all of the other students in my program were preparing for a job in academia, the traditional next step. I was more interested in the application of theory, and ended up starting my own company to produce games and apps. Perhaps I should have stayed in academia long enough to study some of these important and fascinating research topics! The software development community needs answers sooner rather than later, given this technology’s vast potential for positive, or negative, impact.virtual-reality-classroom

A Family Affair – Creating the Trailer for Crayola Color Blaster

I love the trailer for our new Tango game, Crayola Color Blaster. While it doesn’t actually explain how the game works, it communicates a couple of things really clearly – it takes place in your real environment, you have to move around…a lot, and most importantly, it’s fun to play.

When you are a small company like Legacy, you make use of all the resources you have at your disposal. I am happy to report that it was all hands on deck in the creation of Crayola Color Blaster’s trailer.

  • Adam McClelland, daughter Rachel’s long-time boyfriend and talented filmmaker, shot and edited the trailer, dragging his unwieldy and expensive camera equipment to LA from NY.
  • Most of the gameplay footage was shot in my home in Los Angeles. (I should have cleaned up a bit beforehand!)
  • The trailer stars two of my grandchildren, Jude (7) and Teddy (5), who are the first two kids you see. The big kid, Leah, is my grown up daughter who likes to play games like her momma.
  • Our local recreation center, at Highland Park, was very supportive and allowed us to test and shoot 15 kids. Some of the kids are from there. Their gameplay feedback was incredibly important, especially on complicated interface issues.

Anyway, like everything else with this project, creating the trailer was really fun. Thank you everyone!