It’s Goodby Again

Any James Taylor fans out there?

Today is my last day at HitPoint Studios, where I’ve served as president for three years.

What’s it been like working at HitPoint? Exciting, humbling, rewarding. I worked on many deeply satisfying projects during my tenure at HitPoint.

But it’s more than the specific projects and revenue, of which I am justifiably proud. (HitPoint made the Inc 5000 list of fastest growing companies this year: It’s about the people I’ve come to know and admire. I originally thought that HitPoint’s strength was its engineering talent, especially in AR and backend server technology, but now I know that the UI/UX, product management, art and interactive design staff are also topnotch. And CEO Paul Hake is the heart and soul of HitPoint. I thought I knew about managing a small business, but I learned so much watching Paul create a fun, transparent, disciplined, respectful and productive company culture. I’ve said this to numerous clients and it’s true…HitPoint is the best managed game studio I have ever worked with.

Let me highlight just a few of my favorite projects over the past three years:

1. In 2019, HitPoint participated in a contest sponsored by Niantic to build a product on their platform, “to show what’s possible.” Given that I grew up within a stone’s throw of the Gettysburg Battlefield, I was thrilled that our pitch to create an educational, fun app for families visiting Gettysburg was funded. We learned so much about location based, AR, multiplayer experiences, what works and what doesn’t. I’m hoping HitPoint has the opportunity to build on that experience in the near future.
2. MindLabs Energy and Circuits is a NSF and NIH funded magical STEM learning tool for children ages 8 and older. It combines a digital app, physical cards and AR in a fun and exciting approach to learning core science concepts. And it was designed from the get-go for remote, collaborative learning – ideal for virtual education. MindLabs goes on sale October 1!
3. Venture Valley, about to soft-launch in Canada with Super League, is an esports game that exposes players to the fundamentals of entrepreneurship and financial literacy. It’s a gutsy attempt to support high school and collegiate esports leagues with a relevant game that is fun-first but also aligned to their educational mission. Kudos to the Singleton Foundation’s vision for funding such an innovative, amazing product.

So what’s next for me? A friend told me recently, “You’ll have lots of jobs; just make sure the last one is the best.” While I was working for HitPoint, Legacy Games, a company I founded 22 years ago, continued on its merry way selling casual CDROM games to retail. I am returning to Legacy, to help guide it into the future and hopefully build on its core strengths in publishing and distribution. (Of course I also want to spend more time with my growing, and delightful, brood of grandchildren.)

Though I lack the art

to decipher it,

no doubt the next chapter

in my book of transformations

is already written.

I am not done with my changes. 

– Stanley Kunitz, Poet Laureate

I hope you and your loved ones stay safe and healthy.

The Future of Augmented Reality is Multiplayer

As someone who has been thinking about augmented reality and experimenting with different use cases for the past five years, I have come to a conclusion. AR applications in which we can’t share our experience with someone else are not very compelling.

This doesn’t mean that an AR app has to be fully real-time multiplayer to be enjoyable. But it does need to provide a way for us to visualize the virtual experiences of other users. Silos suck, as perhaps we are all coming to realize with social distancing. I am not interested in evolving a technology, namely AR, that cocoons us even further into our own version of reality.

HitPoint has developed three AR experiences in the past year that are multiplayer or team-based by design. Two of them use physical cards, and one utilizes location/spatial AR. I’ve described the Caesar’s beer pong game and Niantic Gettysburg experience previously. Our deepest multiplayer AR implementation, however, is with an educational circuit-building app, in partnership with Explore Interactive and Purdue University.

In the single player portion of the Explore app, students progress through a series of lessons that teach them about energy providers (batteries) and energy users (light bulbs) as well as related concepts such as closed circuits, voltage, switches, etc. Students choose a physical card, each representing a different electrical component, and lay it on a virtual grid on a table. The position of all the component cards is defined relative to the placement of the main character target cards (Atom and Anne), which also serve as a focal point for the AR logic. Students draw wires on their phone connecting the different components, then look through the camera to see the result, e.g., the light bulb lights up in AR when the circuit is complete.

In addition to the directed lessons, there is “create” mode in which students can build whatever type of circuit they want with the available target cards. (Not surprisingly, the first thing most kids do is start a virtual fire by overloading the circuit.) The create mode can be played as a group activity, with up to four kids participating at one time via a network connection. They can take turns with a single device or each have their own tablet. They see what the other children are doing – laying down a card, drawing a wire on their phone, completing a circuit, etc. – as it is happening. This is true whether or not the children are sitting around a table facing each other or playing remotely.

The definition of 3D space in every AR experience is complex but must be solved if players are to see the same thing at the same time. When the AR target card (and local AR space belonging to it) updates position and orientation, the physics of the observed space are expected to follow. This presents particular problems if projectile objects are in motion, such as in our beer pong game where the ball is being flicked from one player to another.  A simple solution is to pre-calculate the trajectory of projectiles so it can be rendered at a known point x for time t. This ensures that both players will see the projectile in the same position without expensive network synchronization, but it lacks the real-time responsiveness that a physics engine can provide. Our solution for beer pong was to define all physics as relative to the AR target card, so even when the target card moved the physics were the same. 

This image has an empty alt attribute; its file name is IMG_0009-1024x768.jpg

The Explore circuit game does not include projectiles, but it has other challenges. Particles (sparks/fires from batteries) are difficult to synchronize because that would slow down the gameplay given network realities. We fake it, however, so that all the players see the same effect at the same time, but not identical graphics. It’s more important that the players get a shared impression of the sparks system rather than an exact replica. Tricks of the trade.

We’ve learned a lot about designing a multiplayer AR game through trial and error. Initially, we used Vuforia to handle image tracking and some plane detection, but we found that if the kids jostled the character target cards or otherwise obstructed the camera’s view of the cards, the entire grid would disappear or flip orientation.  Integrating ARKit and ARCore to track the position of the camera using SLAM added immeasurably to the app’s stability and made multiplayer much less frustrating. By using these advanced AR technologies to create a sense of space beyond the visible target cards we provided a smoother, more enjoyable experience.

Another problem in AR that still confounds us relates to a common multiplayer issue – conflict resolution. What do you do when four kids are manipulating the target cards and drawing virtual wires at the same time? Whose reality do you reflect and how do you communicate that to the players? This is particularly true for connecting and deleting virtual wires between components. When things don’t happen as expected, children assume they have done something wrong or the app is broken when instead, another child grabbed the wire first. We’ve started to implement a communication system using emojis (because who would trust 5th graders with a chat feature?). We’re also experimenting with user interface changes, like blinking wires before they disappear and various sound effects. Especially when a sense of physical reality is attributed to a virtual object, providing noticeable reactions to every action is important.

No one has ever said that multiplayer AR is easy. The technical and user interface challenges are tough and compounded when hardware is old and/or network speeds are poor, a situation commonplace in schools. Nonetheless, students in our testing have been very enthusiastic about the group circuit building experience, further cementing my view that multiplayer AR is the future of augmented reality.

Learning From the Past with Location Based AR

Imagine 6000 acres of meadowlands occasionally crisscrossed with stout wooden fences and narrow roads. Apart from some monuments and statues, there is little here to indicate the crucial role these softly rolling hills in Central Pennsylvania played during the Civil War. Cemetery Hill, Little Round Top, Devil’s Den, Seminary Ridge…on these solemn landmarks the Confederate Army battled the Union Army at Gettysburg for three days, resulting in 50,000 casualties.

All of us at HitPoint Studios wanted to tell the story of what happened at this battle, generally considered the turning point in the Civil War. We had previous experience creating location-specific games, (including a game for Dreamworks and their Train Your Dragon franchise), and felt the time was ripe to combine geolocation, AR, and gamification to create a rich interactive experience at the most-visited battlefield in America.

Then Niantic announced the Beyond Reality Developer contest, and we pounced. It was the opportunity we were looking for. Niantic’s goal was to encourage developers to use and give feedback on their platform while exploring new use cases and gameplay mechanics. As one of ten finalists, HitPoint’s goal was to create an experience at Gettysburg that would engage the almost 1,000,000 annual visitors in new ways. (You’ll have to read to the end to find out if we won!)

Turning those empty fields into compelling interactive moments, while still maintaining its solemnity…it was beyond hard. We started by determining which Points of Interest (POIs) like monuments to include on our virtual map. We used geolocation to track player positions in real time. As the player walked towards one of the POIs, then selected it on the map, they were able to play mini-games and receive badges and points. Players could play solo or collaborate with others on a team to earn a high score on the leaderboard.

Visiting some of the monuments prompted a “citizen archeology” game, in which the player used a fun detector tool to locate pieces of an authentic artifact, e.g., Winchester Rifle. Beeping sound effects and an on-screen meter helped guide the player to the underground item. Once the dig site was located, the player could use a virtual pickaxe and brush to dig up, clean, and assemble the buried artifact.

But a true understanding of Gettysburg can’t be achieved simply by providing information and facts. We thought long and hard about how to invoke the emotion and pathos of the battle. Our solution was to incorporate a soldier on the virtual map who marched along a set route at certain times. If the player moved close enough to him as he passed, he gave the player a letter from an actual combatant. The letter unfolded and was read aloud, with 3D text appearing in AR on the horizon. “A fierce battle was fought here today…”

Once the letter was read, the soldier gave the player a virtual rose. The rose could be placed anywhere as a tribute; its GPS location was saved permanently. Players were able to see their own glowing rose as well as anyone else’s rose at any time.

Evaluating and testing an app when 3000 miles away from the location it was designed for was challenging. We “spoofed” monuments wherever we could around our office, (sidewalks, hallways, etc.) but that obviously wasn’t the same as actually being there. Fingers crossed, we visited the battlefield with the indefatigable Garry Adelman, Director of History and Education at the American Battlefield Trust, and had a blast trying it out with some enthusiastic visitors.

So how did HitPoint do in the Niantic Beyond Reality Developer Contest? We learned so much about the player’s expectations, what they found fun (and not fun), the critical importance of UI in augmented reality experiences, what’s good, (and what’s still baking in the oven), with the Niantic platform, and much more. But sadly, we didn’t win the contest. Not discouraged in the least, we’re now busily planning which historical and cultural locations to focus on next. Stay tuned!

Great Project-Great Partners

We are thrilled and humbled that the National Institute of Health is supporting us with our second Small Business Innovation Research grant. It really is a dream opportunity and team. The corporate leadership at Explore is smart, focused, and energetic. Academics from Purdue University are providing the theoretical framework and research results. And the project targets HitPoint’s sweet spot – challenging, scalable technology. We will be creating a multiplayer, collaborative AR platform with an open-ended style of learning, for teaching science concepts. I can’t wait to get started!

A quick video:

More details here:

How to Fix the Smartphone

Jonah Lehrer, my son and wonderful science writer, recently posted this blog. He gave me permission to copy it here. Between all of us, surely there is someone who can turn this eye-popping research into an actual product!

The astragalus is the heel bone of a running animal. It’s an elegant part of the skeleton, so curved it looks carved, with four distinct sides. It fits in the palm of your hand.

The astragalus is also one of the most common archaeological artifacts, found in ancient dig sites all over the world. The bones have been uncovered in Greek temples and Mongolian villages, Egyptian tombs and Native American cave dwellings. In Breughel’s masterpiece “Children Games,” two women toss astragali in the corner of the painting. They look like they’re having fun.  

The women tossing astragali in Breughel’s “Children Games”

The women tossing astragali in Breughel’s “Children Games”

Why are these small animal bones such a universal relic? The answer returns us to the peculiar shape of the astragalus. Because it has four sides, the bone can be used like dice: when thrown on a flat surface, it turns into a primitive randomizer, injecting a dose of uncertainty into the game. As the science historian Ian Hacking writes, these dice made of skeletons are so ubiquitous that “it is hard to find a place where people use no randomizers.” 

Of course, we don’t throw bones anymore. Now we have more advanced sources of randomness. Just look at slot machines, those money-sucking devices that enchant people with their unpredictable rewards. Although we know the games are stacked against us, we can’t resist the allure of their intermittent reinforcements. 

Or consider the smartphone. If the reward of slots is the rare jackpot, the reward of these devices is the arrival of a notification. As noted in a new paper by Nicholas Fitz and colleagues in Computers in Human Behavior, “In less than a decade, receiving a notification has become one of the most commonly occurring human experiences. They arrive bearing new information from or about a person, place, or thing: a text from your mom, news about Donald Trump, or a calendar invite for a meeting.” The ancients tossed animal bones to experience the thrill of random rewards. All we have to do is glance at these gadgets in our pockets.

There’s nothing inherently wrong with notifications. Unfortunately, their intermittent delivery (and the way they are constantly evolving to become more salient and sticky) creates a digital system that sucks up our attention, which is why the typical America spends 3 to 5 hours a day staring into small shiny screens. The end result is a permanent state of distraction, a mental life defined by its addictive interruptions.

Is there a better way? This urgent question is the subject of that new paper by Fitz et al. The scientists explore the potential benefits of creating smartphone notifications that are batched and predictable, arriving at regular intervals throughout the day. If our current smartphone experience is like a pocket slot machine, every random beep another reward, these batched notifications try to remove the twitchy uncertainty. We know exactly when the rewards will arrive, which will hopefully make them far less exciting.

To test the effectiveness of this setup, Fitz et al. recruited 237 smartphone users in India. Each of the users was randomly assigned to one of four conditions: 1) notifications received as usual 2) notifications batched every hour 3) notifications batched three times a day 4) or no notifications at all. The conditions were implemented using a custom-built Android app.


Which setup worked best? It wasn’t close—batching notifications into three predictable intervals led to improvements across a wide range of psychological outcomes. (Hourly batching was less effective, though it did lead people to feel less interrupted by their phones.) According to the data, those who got three batches reported less inattention, more productivity, fewer negative feelings, reduced stress and increased control over their phone. They also unlocked their phones about 40 percent less often.

Interestingly, silencing all notifications tended to backfire, boosting anxiety without any parallel benefits in focus. (People were still distracted, just by their FOMO, not their gadgets.)

This research comes with enormous practical implications. In a little over a decade, the smartphone has transformed the nature of human attention, consuming gobs of our mental bandwidth. It’s a consumption we often underestimate. According to Fitz et al., most people think they get about thirty notifications per day. The reality is far worse, with the typical subject receiving more than sixty beeps, pings and buzzes. But if you ask them how many notifications are ideal, they give an answer closer to fifteen. In other words, we desire technology with limits, a smartphone that shields us from its own appeal.

And this brings us back to the power of intermittent reinforcement. Randomness has always been entertaining. The difference now is that we’ve engineered a technology that’s simply too irresistible—software evolves far faster than our hardware—which is why we end up spending more time staring at our phones than we do parenting, exercising or eating combined.

But it doesn’t have to be this way. One day, a gadget maker will give people what they really want: a machine that doesn’t hijack the brain. Based on this paper, a core element of this future gadget will be a default notification system that delivers its interruptions in predictable batches. That text can wait; so can the update from the Times and Twitter; we don’t need to know who liked our Instagram in real time.  

Sometimes, less is so much more.

Fitz, N., Kushlev, K., Jagannathan, R., Lewis, T., Paliwal, D., & Ariely, D. (2019). Batching smartphone notifications can improve well-being. Computers in Human Behavior.

Surprise and Delight, Rinse and Repeat

Can you believe this video has garnered almost 1 BILLION views?

Ryan ToysReview is a phenomenon. Since 2014, “unboxing” channels on Youtube have changed the face of children’s television and altered the way kids discover new toys.  Mega star, Ryan, now 8 years old, generated 22 million dollars last year from his unboxing videos and has created a new toy and apparel line for Walmart that includes blind bag collectibles. Nickelodeon even announced a new TV show, called Toy Toy Toy – The Unboxing Show.

The viewer doesn’t know what will be revealed in an unboxing video, but they anticipate that it will be something they like. It is that anticipation combined with mystery that make these videos so compelling.  Humans love the unknown. The neurotransmitter, dopamine, is associated with pleasure of all sorts – sex, drugs, and video games – and delight spikes with uncertainty. Turns out we get bored quickly; the first taste of cake tastes vastly better than the third. Habituation to stimuli is a key attribute of our nervous system. In contrast, if you see something surprising or unknown, your attention becomes immediately engaged. (This effect is obvious even in infants who pay more attention to an “optimally scrambled” face than a normal one.) Neuroscientists call this a “prediction error.” It’s the delight we never saw coming.  It doesn’t matter if it’s a loot box in a video game or one of Ryan’s unboxing videos. These mysteries excite our dopaminergic system, which is why we like them.

Toy creators paid attention to the burgeoning popularity of unboxing videos and two years after Ryan first burst onto the scene, MGA introduced the L.O.L. line of toys. The company wanted to cash in on the unboxing and collectibles trends, and so it came up with more than 250 dolls whose identities are hidden until unwrapped.

L.O.L collectible dolls

How does it work?  The L.O.L. Surprise! Pearl Surprise toy boosts of “3 layers of surprises.” First you open up the six plastic pearls, each containing clothing for the LOL collectible dolls. Next, you unwrap a giant seashell covered with sand, then place it in water. It fizzes for a few minutes, until all of the “sand” falls off. Amidst some lovely pastel colors generated by the transformation, a plastic shell emerges. You open the shell, and there are three “surprise” bags inside containing, finally, the dolls and accessories. The transformations and surprises keep coming. Even the dolls hair and skin changes colors when dunked in water. (The down side is that all this packaging stuff is filling up our landfills!)

Like other similarly successful toys, L.O.L. dolls combine three key elements: (1) Surprise within a known collectible universe. The child can anticipate the type of toy but not the specific one in the collection, thus holding out the promise of obtaining a “rare” specimen; (2) Multiple surprises. The toy is teased through layers of unwrapping, with delight and surprise along the way; (3) Toy transformations. The child is required to take an active role in the physical transformation of the toy, e.g., adding water to the seashell.

And just so you don’t think that these toys are only for girls, here’s another example. Grandson Izzy loves Dino eggs.  He prefers to hack at the egg shell with a plastic pick to reveal the specific dinosaur inside, but you can also submerge it in water and watch the shell fall off in the fizzing water.  Voila, a triceratops!

It’s genius, combining anticipation and mystery with actions that reveal further surprises. By emphasizing the process and making the child feel as if she is contributing to making the product, the child has an immediate sense of ownership. The child is the creator, not just a purchaser. It feeds their curiosity and that strange pleasure that comes from pursuing the unknown.

HitPoint’s Fairy House

How does all this relate to Augmented Reality, my primary interest these days? I believe that AR is the perfect medium for combining the power of mystery and subsequent revelation with a physical toy. At HitPoint Studios, we created a prototype of a toy fairy house and mailbox and then combined it with virtual fairies. The child receives virtual messages from the fairy queen, suggesting mobile AR games and activities to do in and around the fairy house. Over time, more and different fairies appear as the child carries out activities to make the fairies happy.  Likewise, the physical fairy house takes on new virtual features, like furniture, lights and sounds.

HitPoint’s Fairy House combines features of some of the bestselling toys described earlier. The mysteries of the fairy world are revealed gradually, through the child’s actions, thus causing virtual transformations (rather than physical) in the toy and surrounding environment. Surprise and delight are key elements to every interaction. Unfortunately, HitPoint hasn’t successfully sold a toy manufacturer on our vision…yet, but we have high hopes that once a company has success with this approach (LEGO?), more companies will follow.

Multiplayer AR – Good, Bad, Ugly

AR is an impressive technology but until the launch of ARKit 2.0 last year, single-user experiences were the norm. Unfortunately, these early apps neither retained customers, nor monetized. (We know this from first-hand experience!) Since then, multiplayer apps, where people can share a virtual experience in real time, have gained steam. Players can affect the same environment and see different things depending on where they are positioned. Just as in gaming generally, social playing is simply much more fun.

Recently HiPoint Studios partnered with Caesars Entertainment in its first foray into multiplayer AR gaming. We created a series of mini-games for a new bar experience in The LINQ Hotel in Las Vegas. Using the bar coasters as targets, five different AR flick-style sports games (baseball, football, basketball, hockey, and beer pong) can be played, with high scores posted on an online leaderboard.

One of the games, beer pong, is a multiplayer game. And here, let me go straight to the punch line. Both the design and engineering of multiplayer AR games are super challenging but using target-based AR considerably reduces the risks.

Beer pong has a well-understood gameplay mechanic, so designing a two-player version around a bar coaster was not difficult.  However, designing multiplayer AR that is NOT target based can be hard because the players have complete control of their camera and, depending on the gameplay style, may need to know what the other player is seeing and how it is different from their perspective. Imagine a multiplayer AR combat game where your opponent is hidden on the roof, not visible from your orientation? How does a designer encourage player movement, perspective shifting and mental rotation, when necessary? The UX challenges can be monumental.

Regarding technical hurdles for multiplayer AR, relocalization is one of the toughest nuts to crack. Until world maps could be saved across multiple sessions, and transferable between devices, users could not revisit a location and find that the app remembered it. It also meant that AR experiences were generally always solo ones. Then Apple introduced relocalization in iOS 11.3, which let users restore a state after an interruption, allowing them to relocalize a world map to a later session, or share to another user or device. There are some highly complicated visual computing systems to support this process, e.g., SLAM map coordinates, but relying on a common physical tracking marker image or QR code, as we did in the Caesars game, neatly solves the problem.

For the marker to work, both players point their phones at the coaster on the table in front of them; the app treats the marker as the origin coordinates, making the real world and the virtual world consistent across both phones. This works quite well and removes some low-level problems of figuring out where the players are relative to each other. Instead, the device makes assumptions based on where they are relative to the coaster.

Meanwhile, target-based multiplayer AR has an obvious disadvantage. While it works quite well in a specific location like a restaurant or office, only certain kinds of games or experiences lend themselves to playing around a coaster or beacon. It’s definitely a good short-term fix for multiplayer AR, and a great way for our company to “get its feet wet,” but obviously only the beginning.

What’s next for the team at HitPoint, now that the Caesar’s LINQ game has launched? Given our extensive experience with the now defunct Tango phone, our engineers are particularly excited about the Cloud Marker tech from Google that uses recognized landscapes for relocalization (similar to the ADF scans that we used with Tango for some cool AR retail demos.) Can’t wait to see what the HitPoint team comes up with next!

A Year of “Firsts”

Happy Holidays 2018!

Last year around this time I announced that I was joining HitPoint Studios as President, after 20 years at Legacy Games. It was a momentous decision, made easier by the fact that Paul Hake, HitPoint’s CEO, and I had worked together on numerous projects previously.

So how’s it gone? 2018 was, simply put, a year of “firsts,” both for me and HitPoint Studios. Our goal was to challenge ourselves with new, interesting, and technically complex projects. (I think we may have overachieved!)

1 – We launched our first real-time multiplayer mobile game last week, for none other than Ellen Degeneres Digital Ventures. Loosely based on her popular daily TV show, Game of Games, we are proud of the app’s gameplay and polish, and will continue to build on its success next year.

2 – We developed our first  Remote Gaming Server (RGS)  – a cross platform, cloud-base data management platform for casino games, with extensive social features. It is now fully licensed and operational in Europe. We will be adding games to the RGS throughout 2019.

3 – We created our first multiplayer, location-based AR game for Caesar’s.  PlayLinq is a collection of AR sports mini games (baseball, football, basketball, hockey, beer pong) designed to be played in location at the LINQ Hotel in Las Vegas. (BTW – Multiplayer AR is NOT for the faint of heart!)

4 – And switching gears…we designed our first smart toys, starting with an adorable fairy house and virtual fairies, and continuing on with AR controlled vehicles (super-secret project).

5 – We collaborated with Explore Interactive to develop our first educational AR game, designed to teach kids about electricity.  It includes a series of “challenges” to build specific types of circuits, along with an open-ended “create” mode.

And if that doesn’t sound like a sufficient challenge for our 25-person team in MA and CA, in 2018 we also (a) continued work on the Disney Magic Timer app, adding enhanced AR plus a robust Content Management System, (b) launched a beautiful multi-platform adventure game, Adera, (c) developed a second Facebook Instant Game and (d) ported Crayola Color Blaster from ARKit/ARCore to the Lenovo Mirage AR. 

We’ve learned a lot this year, some of which I tried to capture in blog posts here and when speaking at conferences. There have also been plenty of bumps and bruises, as well as some magical moments. Reminds me of playing a well-balanced game, when the next move seems to be within reach, but you have to stretch to get there.

We’ve been stretching a lot this year at HitPoint, and it feels good.

Wishing you peace, good health, prosperity, and your own satisfying challenges in 2019.













Augmented Reality and Schools – What’s Next?

What determines if an EdTech product is successful? The criteria are daunting. (1) It must teach something faster or better than is possible otherwise. (2) It must be relevant to the curriculum and teachers’ needs. (3) It must be affordable and accessible to all schools and students. (4) Ideally, it engages the student through experiential, discovery learning rather than rote memorization.

How hard is it to create this mythical piece of technology? Well nigh impossible, if looking at the wreckage of past innovations is any indication. I’ve written previously that the most successful attempts have been those hardware and software products that could be used by teachers in a flexible fashion, as tools to teach a variety of different subjects. Computers, tablets, Hypercard, Microsoft or Google office software, programming languages, graphics apps, etc. all make the list of successful technology innovations in classrooms. Their precise level of success, of course, depends on how well they are implemented in the classroom.

Now Augmented and Virtual Reality are headed for the classroom. How are they faring? It is difficult to justify the use of VR in the classroom, given the cost of most hardware. And while Google Cardboard is affordable, it is limited to presenting pre-rendered visual experiences in 3D. Is it that much more effective than watching a YouTube video? I doubt it.

And what about Augmented Reality apps for the classroom? I’m admittedly more of a fan of AR than VR. In its best incarnation, AR incorporates the real environment as part of the experience rather than create an alternative world as in VR. Some types of AR are possible with low end mobile phones, so cost isn’t necessarily an issue.  However, judged against the success criteria I listed initially, AR is sorely lacking.  It is not clear that most of the curriculum based AR experiences are more effective than other teaching methods.  And while there are some open-ended tools for creating AR experiences (e.g., Blippar, Zappar), they are focused on the consumer market, and not geared to school use. Eon Reality and Jig Workshop both have an authoring tool for teachers, but they are limited in what kinds of interactions are possible with pre-packaged 3D objects.

Meanwhile, AR is being rapidly implemented in industry, with use cases that would be equally helpful in K-12 schools and colleges.  Businesses are challenged with training millennials to take the place of retiring Baby Boomers, and they are finding that Augmented Reality is the ideal tool for any kind of procedural learning. Workers are  taught how to fix or assemble complicated equipment using step by step procedures that appear directly on machines. Similarly, workers are learning advanced manufacturing techniques, with operational guides turned into 3D displays.  Real time collaboration as well as asynchronous communication are possible using AR, with the remote expert providing help and guidance by leaving messages or drawings in the relevant location for workers.

How would that type of functionality fit into the classroom? Well clearly (and I’d like to be on the team that creates this), this mythical app would need to be designed specifically for schools, and include all the administrative, networking, and privacy functionality required. it would greatly facilitate distance learning.  A remote or in-classroom teacher could look at what a student is working on, e.g., 3D printing, robot assembly, chem lab, shop, frog dissection, circuit builder, etc. and write or speak comments, displaying instructions directly on the objects in question, in real time or asynchronously. Alternatively, two or more students could work collaboratively, in the same location or remotely, using this AR technology.

Some innovative developer is going to take the collaborative AR functionality being used successfully in industry today, and build a similar platform for high schools, technical schools, and colleges to engage the learner in new ways. I’d love to see a STEM-oriented tool for students to practice the engineering design process from problem to prototype while seamlessly collaborating and troubleshooting with the teachers and peers using AR.  It’s coming, and I, for one, can’t wait for the future to get here!

Augmented Reality in the Classroom – Lessons Learned

Many companies with glitzy AR apps have come and gone; many educational use cases have been tried and discarded. Is Augmented Reality just the latest in an endless stream of educational technologies that don’t add up to more learning?

Maybe. Beautifully rendered virtual images of the periodic table and human anatomy or spinning globes have failed to become standard features of the school curriculum. Volumes have been written about why, but it comes down to cost, convenience, and ROI. Educational AR apps tend to be difficult and costly to create and must compete with traditional “supplemental” school teaching aids. Students generally have to look through an expensive smartphone to see the virtual graphics and animations, making the interface awkward. While the initial experience engenders surprise and delight, there is little need, or desire, to repeat the experience. It’s also not clear how seeing a dinosaur “come to life” from the page of a textbook actually increases learning.

If you look at the history of educational technology, the technologies that ultimately earn a place in classrooms are used as tools first, rather than to directly deliver curriculum content. Consider Apple’s Hypercard, introduced in the late 1980s. Teachers loved it and similar products because they were easy to integrate into any subject matter. They turned kids into active creators rather than simply consumers of content. Plus they exposed children to more “real world” uses of technology, better preparing them for future jobs. Tools like Hypercard did more than anything else to ensure a permanent place in classrooms for computers.

The first interactive product I ever produced was Children’s Writing and Publishing Center, published by The Learning Company. It allowed kids and teachers to easily create newsletters, brochures, etc., similar to PrintShop but specifically geared to educators. It was a huge hit. My next product was another language arts tool, Mickey’s Crossword Puzzle Maker. Using Disney characters, kids could create and print out their own picture-word crossword puzzles. It was already clear, back in the 1990’s, that open-ended, tool based learning was the best way to insinuate technology into schools.

There are, of course, many tools designed to help educators create Augmented Reality content and experiences. The more popular AR authoring tools that don’t require programming are Zappar, Blippar, HP Reveal, and Metaverse. But even the simplest of these AR creation tools are made more difficult because they require the ability to make, access, and manipulate 3D graphics. (That’s the virtual part!) There is simply not a great option for K-12 schools, i.e., one that is easy enough to be used by teachers and students but sufficiently sophisticated to support mixed reality experiences that go beyond simple target based 3D pop-ups.

My sense is that the path that Google has chosen to win teacher acceptance for its AR/VR technologies will turn out to be the most productive. Google Expeditions is a great start, but students are still consumers of content, not creators. However, with Tour Creator, Google’s latest AR/VR product for schools, students grab scenes from Google Street View, and compile them into a Powerpoint-like presentation with text and voice. Neatly solving the dilemma of where to get and how to manipulate 3D assets, Google’s Poly and Street View provide 3D graphics and 360 video for most locations. Soon, Google says, students will be able to view their tours created with Tour Creator in their Expeditions app and on new Chromebooks that now include ARCore, Google’s AR platform.

But there is much more that can be done. Why not…

  • Add a story creator capability to Tour Creator, so students can add more and different kinds of assets and content to their tour?
  • Create open-ended AR tools that are designed for specific subject areas, like chemistry or physics? Augmented reality could be used to highlight the reaction when you mix together chemicals to form a solution.
  • Design AR tools for real-time remote communication? Businesses are rapidly implementing AR to train workers on the repair of complex equipment. Why not use AR similarly to help students building projects in Maker Labs?
  • Develop an AR game creator with a variety of templates that can be customized or a quiz generator that uses object recognition to teach foreign languages?

We’re at the Hypercard inflection point with Augmented Reality. Now we just need the right tools to convince teachers that this is an educational technology worth having.