As someone who has been thinking about augmented reality and experimenting with different use cases for the past five years, I have come to a conclusion. AR applications in which we can’t share our experience with someone else are not very compelling.
This doesn’t mean that an AR app has to be fully real-time multiplayer to be enjoyable. But it does need to provide a way for us to visualize the virtual experiences of other users. Silos suck, as perhaps we are all coming to realize with social distancing. I am not interested in evolving a technology, namely AR, that cocoons us even further into our own version of reality.
HitPoint has developed three AR experiences in the past year that are multiplayer or team-based by design. Two of them use physical cards, and one utilizes location/spatial AR. I’ve described the Caesar’s beer pong game and Niantic Gettysburg experience previously. Our deepest multiplayer AR implementation, however, is with an educational circuit-building app, in partnership with Explore Interactive and Purdue University.
In the single player portion of the Explore app, students progress through a series of lessons that teach them about energy providers (batteries) and energy users (light bulbs) as well as related concepts such as closed circuits, voltage, switches, etc. Students choose a physical card, each representing a different electrical component, and lay it on a virtual grid on a table. The position of all the component cards is defined relative to the placement of the main character target cards (Atom and Anne), which also serve as a focal point for the AR logic. Students draw wires on their phone connecting the different components, then look through the camera to see the result, e.g., the light bulb lights up in AR when the circuit is complete.
In addition to the directed lessons, there is “create” mode in which students can build whatever type of circuit they want with the available target cards. (Not surprisingly, the first thing most kids do is start a virtual fire by overloading the circuit.) The create mode can be played as a group activity, with up to four kids participating at one time via a network connection. They can take turns with a single device or each have their own tablet. They see what the other children are doing – laying down a card, drawing a wire on their phone, completing a circuit, etc. – as it is happening. This is true whether or not the children are sitting around a table facing each other or playing remotely.
The definition of 3D space in every AR experience is complex but must be solved if players are to see the same thing at the same time. When the AR target card (and local AR space belonging to it) updates position and orientation, the physics of the observed space are expected to follow. This presents particular problems if projectile objects are in motion, such as in our beer pong game where the ball is being flicked from one player to another. A simple solution is to pre-calculate the trajectory of projectiles so it can be rendered at a known point x for time t. This ensures that both players will see the projectile in the same position without expensive network synchronization, but it lacks the real-time responsiveness that a physics engine can provide. Our solution for beer pong was to define all physics as relative to the AR target card, so even when the target card moved the physics were the same.
The Explore circuit game does not include projectiles, but it has other challenges. Particles (sparks/fires from batteries) are difficult to synchronize because that would slow down the gameplay given network realities. We fake it, however, so that all the players see the same effect at the same time, but not identical graphics. It’s more important that the players get a shared impression of the sparks system rather than an exact replica. Tricks of the trade.
We’ve learned a lot about designing a multiplayer AR game through trial and error. Initially, we used Vuforia to handle image tracking and some plane detection, but we found that if the kids jostled the character target cards or otherwise obstructed the camera’s view of the cards, the entire grid would disappear or flip orientation. Integrating ARKit and ARCore to track the position of the camera using SLAM added immeasurably to the app’s stability and made multiplayer much less frustrating. By using these advanced AR technologies to create a sense of space beyond the visible target cards we provided a smoother, more enjoyable experience.
Another problem in AR that still confounds us relates to a common multiplayer issue – conflict resolution. What do you do when four kids are manipulating the target cards and drawing virtual wires at the same time? Whose reality do you reflect and how do you communicate that to the players? This is particularly true for connecting and deleting virtual wires between components. When things don’t happen as expected, children assume they have done something wrong or the app is broken when instead, another child grabbed the wire first. We’ve started to implement a communication system using emojis (because who would trust 5th graders with a chat feature?). We’re also experimenting with user interface changes, like blinking wires before they disappear and various sound effects. Especially when a sense of physical reality is attributed to a virtual object, providing noticeable reactions to every action is important.
No one has ever said that multiplayer AR is easy. The technical and user interface challenges are tough and compounded when hardware is old and/or network speeds are poor, a situation commonplace in schools. Nonetheless, students in our testing have been very enthusiastic about the group circuit building experience, further cementing my view that multiplayer AR is the future of augmented reality.