Is Tango the Future of AR?

The much anticipated first Tango-enabled device is finally released to the public, along with Legacy’s new Augmented Reality game, Crayola Color Blaster. Where do we go from here?

promo_2For Legacy, the answer is easy. We have new content for Color Blaster releasing Valentine’s Day 2017. Instead of running away from zombies, you’ll have to avoid rampaging gnomes, orcs, and dragons. Seriously, we will be focusing most of our future efforts on evolving the types of interactivity possible with virtual characters, given our discovery of how much fun this is in AR.

unnamedWhat about other Tango apps? Phantogeist is another great game with roughly humanoid shaped aliens invading our world, along with monstrous creatures of different sizes! Both Woorld and Raise bring cute little characters and objects into your room. As the Tango technology becomes better and smarter, virtual characters will be able to recognize what objects are. Imagine virtual humanoids relaxing in your favorite armchair, digital pets sleeping in your bed, a 3D chef inspecting your kitchen and pots, and more!

hotwheelsAnother thing Tango devices do well is the ability to track your movement in space. This helps construction games in particular. Placing straight tracks, curves, accelerators, and much more is intuitive in the Hot Wheels Track Builder because you can pick up the objects and walk it over to exactly where you need it. The Towers for Tango game lets you build skyscrapers on your coffee table. Move your device up and down each floor and get an X-ray view to customize things exactly the way you want.

So back to the original question…Are Tango-enabled smart phones the future of AR? Or is looking through a phone as you walk through a room or building just too wonky to become mass market? Pokemon Go would indicate otherwise. There will certainly be more Tango smart phones to hit the market in the next year, including some announced at CES in January. (Just google it!)

On the other hand, even if Tango-enabled smart phones don’t explode in popularity, it doesn’t take insider information (of which I have none) to predict that Tango software may become embedded in Google Daydream VR headsets. Once that happens, Daydream will become a mixed reality platform (it currently just supports VR), and its form factor will just keep getting smaller and more comfortable.

My point is that I believe Tango software and 3D camera technology is here to stay. Its 3D mapping usefulness, fun factor, and relatively low cost will make it ubiquitous, eventually. The form it takes, however, will vary, at least in the short term. Mobile phones now…headsets next…glasses later? What do you think?

screen-shot-2016-09-24-at-09-39-06-796x364

6 Research Questions for AR/VR

We know surprisingly little about the long term effects of Augmented and Virtual Reality, on both kids and adults. The technology, thanks to large investments of resources and talent, is proceeding faster than our understanding of how to harness it or its consequences. Software developers and hardware manufacturers desperately need the insights obtainable only through verifiable research, before establishing best practices and policies.

chicago-toy-fair

Don’t get me wrong. I am an unabashed technology enthusiast. I believe that AR/VR offer the most exciting new business opportunity since the invention of smart phones. Legacy just completed our first major AR project, built for new “world-sensing” Tango devices. Yet producing Crayola Color Blaster on this remarkable new mobile platform has raised many cognitive and developmental questions for our team. Unfortunately, there is a dearth of reliable, valid research to guide our many design questions.  For example:

  1. How does a VR experience affect, if at all, the child’s developing brain and vision? What is the long term neurological impact of the two small LCD monitors and lenses, each projected at one eye, creating a stereoscopic effect and giving users the illusion of depth? Some hardware manufacturers have established age recommendations, but we already know that children are intensely interested in VR and will be using the devices regardless.making-immersive-virtual-reality-possible-in-mobile-27-638createstereoscopicimages
  2. In a world with unlimited “virtual garbage” potential, how do we direct the user’s attention towards relevant data and away from irrelevant stimuli? What does cognitive load research tell us about how to manage limited attentional resources? This may be our toughest design challenge. 571534199
  3. What are the societal implications of a world that has been customized to reflect our interests? What if our experience of the world is personalized through AR glasses, much as our Facebook feed is currently? Will this just further our descent into silos of interests and beliefs?
  4. How can we create more authentic digital tests of student or employee ability using AR/VR? What if, instead of problem solving tasks delivered via a computer or tablet, we embed virtual tasks in the real world instead? Can we build assessment tools that truly capture someone’s range of talents?
  5. Does memory work the same way for virtual objects as it does for real ones? In our AR game, Crayola Color Blaster, children seemed to have a hard time remembering the placement of virtual items in the room. If memory for virtual objects is worse than for physical objects, we will have to add more spatial cues to AR applications and design around that deficit.
  6. Can we operationalize a “sense of presence,” which many people believe results from an immersive VR experience? Are there corollary physiological measures? Does it lead to increased empathy, as some researchers maintain? Does this also imply that violent, FPS VR apps will have an outsized impact on our emotions?

I was a graduate student 30 years ago in Cognitive Psychology. Almost all of the other students in my program were preparing for a job in academia, the traditional next step. I was more interested in the application of theory, and ended up starting my own company to produce games and apps. Perhaps I should have stayed in academia long enough to study some of these important and fascinating research topics! The software development community needs answers sooner rather than later, given this technology’s vast potential for positive, or negative, impact.virtual-reality-classroom

A Family Affair – Creating the Trailer for Crayola Color Blaster

I love the trailer for our new Tango game, Crayola Color Blaster. While it doesn’t actually explain how the game works, it communicates a couple of things really clearly – it takes place in your real environment, you have to move around…a lot, and most importantly, it’s fun to play.

When you are a small company like Legacy, you make use of all the resources you have at your disposal. I am happy to report that it was all hands on deck in the creation of Crayola Color Blaster’s trailer.

  • Adam McClelland, daughter Rachel’s long-time boyfriend and talented filmmaker, shot and edited the trailer, dragging his unwieldy and expensive camera equipment to LA from NY.
  • Most of the gameplay footage was shot in my home in Los Angeles. (I should have cleaned up a bit beforehand!)
  • The trailer stars two of my grandchildren, Jude (7) and Teddy (5), who are the first two kids you see. The big kid, Leah, is my grown up daughter who likes to play games like her momma.
  • Our local recreation center, at Highland Park, was very supportive and allowed us to test and shoot 15 kids. Some of the kids are from there. Their gameplay feedback was incredibly important, especially on complicated interface issues.

Anyway, like everything else with this project, creating the trailer was really fun. Thank you everyone!

A Career Highpoint – Crayola Color Blaster Launches Today!

untitled-1When you’ve been doing the same thing for as long as I have, 30+ years, you have many career highs and lows. For example:

Highs:  Emergency Room, Law & Order, Murder, She Wrote, Crayola DJ games

Lows: Great Recession/losing our credit line, missing mobile, Twilight Zone

As you might imagine, each of those examples have stories attached, some of them painful. My point is that it is unrealistic to expect one’s career to just go up; my personal trajectory has been anything but.

That’s what makes today so sweet. Legacy’s latest game, Crayola Color Blaster, is launching in the Google Play Store. It was developed as part of the Google App Incubator Project; I can’t say enough good things about the support we received from Google. The app is specifically designed for Tango-enabled Android smart phones, like this Phab 2 Pro from Lenovo.

rocker

Designing a game for a new hardware platform, with innovative 3D depth sensing technology, is incredibly challenging. The interface, i.e., a “magic window” into the world, is completely unknown to consumers. Even more daunting, we had no idea what was actually fun. The game design, originally conceived of as a coloring book in the real world, morphed into a zombie color blaster fairly quickly when we realized that people loved interacting with virtual characters.

scientist

We had an amazing team for the task. My role on the project was as Executive Producer. As such, I assembled the team, managed the relationship with Crayola (very supportive and helpful folks), and gave many “notes” to the production staff. Legacy’s wonderfully creative producer and designer, Andrew Duncan, combined with the rather miraculous engineering of HitPoint Studios and artistic capabilities of Sixth Gear Studios, all came together for a wild six months of constant experimentation and failures, with a few striking successes, culminating in our launch today.

screenshot_20161012-171114

How lucky am I, to be able to do what I love for so many years, and then to experience this kind of career high? It’s the career arc that everyone dreams of, and energizes me to go out there and find the next challenging project. Any suggestions?

 

Polish, Polish, Polish

Many years ago, I heard Doug Carlston, CEO of Broderbund, talk about his approach to product development. I listened avidly, given that Doug was a much admired leader of the leading PC consumer software company at the time, publisher of Carmen Sandiego, Myst, Kid Pix, Printshop, and many other products that I loved.

What really struck me was Doug’s emphasis on the last 10% of product development, the “polish” phase. “It’s really hard. By this time your team is exhausted, everyone just wants to get the product finished and out the door and get their life back.”  But, he went on to say, “This is exactly the time that everyone’s energy and commitment is most critical. Just when the team has the least to give, that’s when they need to give the most.”

promo_1I often think about what Doug said, especially now that we are in the polish phase with Crayola Color Blaster. This is Legacy’s first augmented reality game, to launch at the same time as the Lenovo Phab 2 Pro, the first Google Tango device to enter the market. We just passed our beta milestone, and have about 6 weeks to finish a game that I have fallen in love with. What’s left to do? Feels like everything, examples below.

  • Voice overs and sound – Need to balance sound levels, adjust timing and conditions of voice input, add new music at dramatic moments, etc.
  • Beginning and ending of the game – Need title sequences, credits, celebration at game conclusion, promotion of next game chapter (coming 2017)
  • Disaster recovery/help – It’s new technology, and predictably, doesn’t work every time. What is the user experience when that happens?

All of these and more “polish” items are underway at the same time that our game is going through massive quality assurance and useability testing. Crayola Color Blaster is designed for ages 6+ and designing any app for children, much rather one for a technology that hasn’t even been released yet, has special challenges. Here are just a few of the things we have learned so far in testing.

  • The specific Tango device we are designing for is a cross between a phone and tablet. The 3D and infrared cameras are located on the upper back. We originally designed our game for landscape mode, but discovered quickly that many kids inadvertently cover the cameras with their fingers when tapping the screen to blast the zombies with color. We had to redesign the interface and gameplay to adapt to however the child chooses to hold the phablet in their hands, either landscape OR portrait.

lenovo-phab2-pro-phone

  • One of my favorite features of Crayola Color Blaster is that it is a “walk around” game. You are constantly moving, trying to stay away from the zombies so they don’t “crunch” your color and, if they do, finding paint buckets to suck up new color. This all works well assuming there is at least a 10’ x 10’ clear space in which to spawn new virtual zombies and ambient 3D graphics like flowers and rocks. But what about other environments without the requisite space? We had to design an “Endless Arcade” mode, in which the zombies come toward you from every direction, and you simply need to turn in a circle in order to color blast them.

Quality assurance in general is considerably more complicated with an augmented reality style game, given that it needs to play correctly even as it plays differently, depending on the physical environment. We’ve played the game everywhere we can think of, from elevators (my favorite) to basements to closets. Even so, I’m confident that many enterprising 8 year olds will foil our best testers, see below.

20160825_172321

It must be obvious by now that my wonderful team members and I will be very busy for the next few weeks, trying to add the level of polish that our first Tango game needs and deserves. I see many late nights ahead of us. Wish us luck!

20160825_173125

Crayola Color Blaster at Google’s Indie Game Fest!

Google nominated our AR game, Crayola Color Blaster, for their first annual Indie Game Festival, this Saturday, in SF.  There are two rounds of voting, with the final 5 games winning awesome prizes.

Among the 30 contestants, Legacy has the only game for kids (currently in beta), using a technology that’s not out there yet, for a hardware device that hasn’t shipped. Hmmm…

Wish us luck!!

Interacting With Virtual Characters in AR, and Loving It!

scary-zombie-imgAs a cognitive psychologist as well as game designer, I am intrigued with how traditional play patterns are evolving, given digital games like Pokemon Go that take place in the real world. Add sophisticated augmented reality to the mix, where virtual characters know where they are located vis-à-vis everything else, and it’s a game changer.

Think about it. What if, instead of simply collecting a Pokemon character, you could play tag with Pikachu? What if Haunter actually hid behind a rock and shouted “boo” as you walked by?  What if you were in a race with Growlithe to see who could get to the next Pokemon gym first? Given these kinds of real world, intelligent interactions, Pokemon characters would be dramatically more engaging than their 2D, screen-bound, counterparts. When virtual characters play along with us, we can reimagine almost every traditional game and gameplay pattern, from football to board games to dramatic play.

pokemon-go-google2
Charizard hides behind a real life rock in Google’s original April Fool’s video, a feature not present in the final game.

Why haven’t we seen this type of gameplay yet? A simple reason. The technology that supports it isn’t out in the marketplace yet. Before a 3D virtual character can interact with a game player, the avatar must know where it is in space. This kind of character “intelligence” requires 3D cameras and software that can scan an environment and learn the location of everything, including the game player. (A virtual Pokemon character can’t actually sit on a real park bench until it knows that an object with a particular shape exists in the real world.) Not until the Lenovo Phab 2 Pro launches this Fall, will there be a mass market phone that has the requisite Tango software and 3D cameras that can provide sophisticated augmented reality.

My company, Legacy Interactive, is developing an AR game for devices with Tango.  Since our licensing partner is Crayola and our app is geared to ages 6 and up, we originally thought about designing an AR coloring book. It was fun to color virtual 3D objects like trees and flowers appearing in actual room locations, but it quickly became obvious that this only scratched the surface of what is possible.

20160825_174928We thought about variations of games like tag, freeze tag, capture the flag, hide and go seek, scavenger hunts, etc. and how our coloring book app could be more interactive with the addition of virtual characters. The story line practically wrote itself! A mad scientist takes away all the color in the world, and sets wave after wave of colorless 3D zombies after you. Your task is to color blast them first, before they “tag” you and “crunch” your color, forcing you to find new color buckets and replenish your color.  Meanwhile, the zany professor taunts you between each wave; you have to ultimately find and chase him in the big boss fight.  All of this takes place inside an actual room, while you run around frantically, trying to avoid marauding zombies, and find new paint and special objects to appease them.

Screenshot_20160902-163317What are some other ideas for interacting with virtual characters in our upcoming Tango app?

  • The virtual character wants an object that is hidden in the real world. You must use logic to figure out where it is located and bring it to him.
  • Like color resist, you have to color blast your entire environment in order to “reveal” the virtual objects and “hidden” characters that have been there all the time.
  • Coloring special virtual objects placed in the environment, perhaps combined with a specific color, e.g., rainbow, sparkles, unlocks a special power up,
  • Change your room into new game environments by walking through a magical door or completing a puzzle.  “Poof!” We’ve re-skinned your bedroom with a whole new look and new gameplay parameters, allowing games to progress through a series of scenes and characters. Your room first appears as a dungeon, where you battle orcs, then a spaceship where you befriend little green aliens (my personal favorite).

screenshot_20160902-170508It’s hard to describe, until you’ve actually experienced it, how much fun it is to run around a room, trying to “tag” a virtual character before they tag you. And the more intelligent they seem, by changing their behavior according to where you are in the room or what physical objects are present, the more satisfying the interactions. With this first iteration of Tango, the device only knows that there is an object present of specific dimensions. Eventually the software will be able to identify what the object actually is, e.g. a ball, a table, a refrigerator, and how it can be used. This will open up even more types of interesting interactions with virtual characters.

I can’t wait. This is an extraordinarily exciting time to be designing games. The combination of real world locations, physical movement, and virtual 3D objects and characters that know where you are, provide a rich tapestry of opportunity for game designers.

Look for Legacy’s new AR game, Crayola Color Blaster, in the Google Play store this Halloween!

 

Emerging Play Patterns With Augmented Reality – What Works and What Doesn’t

How does the nature of play change in a world with augmented reality, when your device knows where you are and everything around you?

Pokemon Go broke through gaming’s summer malaise in a spectacularly dramatic fashion, due to its revered brand, fun collector-style gameplay, and “gee whiz” location-based augmented reality integration. But one of the most interesting aspects of Pokemon Go is, in my opinion, how it provides a glimpse into unique play patterns that are enabled by new technologies. Where do we go from here?

The Play Observation Scale (POS) is a good general purpose framework for thinking about the types of play and how it changes over the course of child development: functional (characteristic of very young children), construction, exploration, dramatic, rough and tumble, and games with rules. Within this, play can be solitary, parallel, or group (with one common goal).

stateofplay

Let’s focus on construction play first. (I will address other play patterns in subsequent blog posts.) Traditionally, this refers to assembling blocks, sewing doll clothes, building a fort, and more. Any time a child is a “maker” with an objective, that is considered constructive play. What happens when a child builds something with virtual items? Given AR possibilities due to 3D powered technologies like Google Tango, we now have examples of virtual items actually interacting with physical objects to create an entirely new experience. For example, in Woorld, you can add a spigot to a table top, turn it on, and watch your room fill all the way up to the ceiling with virtual water. To empty the room, you simply add a virtual drain to the floor and watch the water level go down. So much fun!

app-woorld

Here’s an idea. What if there were an AR app where kids power their own virtual machine capable of creating magical virtual objects of any shape and size? Once the machine has been constructed to the player’s satisfaction, a simple touch of a button or lever shoots out a stream of… what? You won’t ever quite know until you try. What will come out the end of the machine will be related to the components that have been added and used previously. For example, let’s say you added a Chicken Button, a Time Machine Rewinder, a Color Randomizer and an Anti-Gravity Extruder, which results in…flying purple eggs! You can move, drop, throw, and scale all the virtual objects created, filling up your room to the ceiling.

thingamajig

This game idea sounds like a dream to someone like me, who for 30+ years has championed the design of products that support open-ended constructivist theories of learning.  Of course it would be even better if the child could create some of the components and not have to rely on just mixing what is in their digital catalog. That would be “Crazy Contraptions” on steroids!

What are the benefits of adding AR to construction play, and is there a significant difference between constructing with virtual objects compared to real ones? Digital items can be easily edited and changed, unlike physical objects. Size them up or down and place them anywhere…even on the ceiling! They are vastly more adaptable than physical objects and allow for creative combinations not possible with real objects.

The disadvantage of virtual objects, and it’s a big one, is that there is currently no haptic feedback with mobile AR devices like Google Tango and precious little with VR either. You don’t feel as if you are actually touching something; you don’t have the experience of twisting and manipulating the object. You can’t feel the screw loosen with the screwdriver, or feel the hammer’s impact on your arm. For children, who learn initially through direct sensorimotor experience with the world, this is a significant issue. The younger the child, the bigger the problem, although the lack of haptic feedback in AR/VR, especially when the experience is mechanical in nature, impedes learning at every age.

lego_75102_box1_na_1488

At this stage of the technology’s evolution, assembling your new Star Wars Lego X-Wing Fighter with real blocks, compared to virtual ones, would be easier and more satisfying. But the finished product couldn’t launch into hyperspace, lost in clouds of stars and planets, like it could in AR. Perhaps the answer is a combination of physical toy with AR? To be continued…

Augmented Reality Run Amok

As a game developer actively working on various AR platforms, I have been mesmerized by the success of Pokemon Go. (And so has everyone else in our business.) Given the number of Pokemon Go knock-offs already in development, there will be many more outdoor, GPS-powered augmented reality games launched soon.

pokemon-go-list1-1200x682

At a recent meeting with my colleagues at Legacy, we were discussing some of the implications of Pokemon style games for advertisers.  The market for virtual advertising, which is almost nonexistent now, will grow exponentially as more AR games and apps that allow you to superimpose virtual images on real objects, indoors and out, enter the market.

I started to riff about how everyone will sell the digital rights to their front yards for virtual billboards. Because of my home’s close proximity to the entrance of Griffith Park (the largest park in LA), will my property be worth more in the future due to all the foot traffic? Patrick, a Legacy employee who enjoys poking holes in my arguments as a matter of principle, quickly dispensed with my Gold Rush dreams.  He shared this dystopian trailer of the future, where EVERYTHING in our environment is polluted with “helpful” avatars and advertising, thanks to augmented reality. Tell me if you don’t feel like poking your eyes out after watching this short film. https://vimeo.com/166807261

571534199_1280x720

Then, to top it off, another Legacy employee, Andrew, reminded me of a famous science fiction short story, The Subliminal Man, by J.G. Ballard.  It is a scary depiction of a consumer-obsessed culture in which subliminal marketing messages are blasted nonstop at unsuspecting workers. I won’t spoil the horrifying ending, but I’ve already hinted at it.  https://biblioklept.org/2013/11/29/the-complete-short-stories-of-j-g-ballard-fifth-riff-the-subliminal-man/

How can we prevent this possible, utterly repellent future? It’s going to require years of discussion and negotiation between technology companies, advertisers, consumers, and government. Can we use hard-won lessons in water, air, noise pollution oversight and policy to avoid polluting our virtual environment? It’s up to us and time to start the conversation.

EMGNpollution2

Video Games Are Good For You!

IMG_3168Guest Blogger, Nicholas Maryan, is an awesome summer intern at Legacy Games.

I’ve played video games my entire life. My mom used to yell at me frequently for “wasting my time.” Yeah, I could have spent a few more hours a week studying my multiplication tables or actually reading that book for my report. Little did my mother know or appreciate that all that time I was working on other important skills.

As a result of playing games, my manual dexterity improved significantly, as well as my critical thinking, decision-making, and ability to type (imperative to efficiently writing papers and emails). In fact, as a result of playing World of Warcraft, I was top of my class in WPM as well as winner of my 5th grade spelling bee. Words like “pristine” and “ogre” were easy for me by that time. And years of playing adventure-based games enhanced my critical thinking skills, forcing me to constantly analyze where to go next (hint: always follow the enemies), strategize how to defeat bosses, or traverse difficult terrain. Games like Call of Duty, in which much of the time is spent playing against other live opponents, help develop quick decision-making skills by anticipating and reacting to others’ moves. Though the specific content (e.g., combat), may not be useful in the real world, the ability to process information quickly and with precision is a vital skill.

WoWphotoforblog
Research has shown that the act of playing video games can improve various skills the more people play. A few studies found that those who play video games develop improved visual attention strategies (i.e. multi-tasking) and are better at analyzing entire scenes and situations rather than one specific object.

Video games, particularly puzzle games, have been found to increase positive emotion, reduce anxiety, and promote overall relaxation. It goes beyond de-stressing as well. There are a slew of mobile games designed to exercise young children’s creativity, dexterity, vocabulary, math, and even cooking skills (just check out Toca Kitchen). These are benefits of the classic console, computer, and mobile games. But what about the new age of Augmented and Virtual Reality?

For those who are new to those terms, augmented and virtual reality are pretty similar. Virtual reality is a completely immersive experience (you have probably seen the headsets) which entirely takes over your visual and auditory senses. Augmented reality does exactly what it says it does, i.e., it augments reality. It imposes digital images onto the real world; the most pertinent example currently is Pokemon GO. Now you can catch “real life” Pokemon from the comfort of your own smartphone.

PokemonGoblogphoto
Since the franchise has been successful for twenty years, Nintendo has captured perhaps the largest age range of any other game in the market today (the nostalgic 30-&-up, down to young kids). The staggering usage numbers are great for Nintendo and the developer Niantic, but this revolutionary game also presents health benefits that could change the industry.

According to one study done by Oppezzo M. and Schwartz, D.L., (2014) participants who walked more, especially among nature, tended to be more creative in their flow of ideas. Another study done by Bratman, G.N. et al. (2015) looked into the effects of urbanization and mental health, and found that those who walk around in nature for an extended amount of time have less negative thoughts on average. Pairing that data with the ability to interact with a mobile video game can have endless positive effects, and learning opportunities. This is where Pokemon GO shines.

Of course, it has also encouraged some bad habits such as blindly walking around lost in your phone screen, resulting in folks falling in a pond, finding a dead body, and crashing cars. Of those who suffer from depression, some claim the app motivates them to get up and get out of the house. In fact, the other day a girl I went to high school with, who suffers from severe depression due to chronic pain, posted a Facebook status celebrating the fact that she finally had a reason to spend time outside.

This alone shows Pokemon’s true potential. No longer is AR seen as simply a more complex and innovative way to play games. It can be a new tool in fighting depression and other mental diseases silently undermining our society. Augmented reality done right can help encourage more interaction between the outside world and gamers of all types, including folks who suffer from mental illness.

As for video games in general, they aren’t all that bad. Video games can even be beneficial for children when used in moderation. I am excited to see what the future of video games holds. One thing I know for sure is that the future is bright, and full of pixels. Have I convinced you yet, Mom?

Nicholas Maryan is a Senior at Kenyon College and is studying Economics. He plans to graduate Spring of 2017. Hire him!