smallminddesign.com

Gestural Interfaces

Anyone who’s seen Steven Spielberg’s 2002 sci-fi movie Minority Report surely walked away from the theatre with his or her heads spinning. Not only for the fantastic action sequences or mind bending story that yours truly is still reeling from. But also the amazing glimpse into the future that Spielberg masterfully weaves into so many of his films with Minority Report being right at the top of that list.

The first time Tom Cruise donned those special gloves to interact with that amazing super information interface, reaching in grabbing windows and tossing them around the screen, had many of us holding our crotches in geeky giddiness and a few in future shock I’m sure. Either way, it was a tantalizing look into what could be with regards to the way we might interact with machines, in particular graphical interfaces. That movie was set in 2054, I surely didn’t want to wait that long to have my own super computer that I could slap around and make neat things happen. But here we are just 7 years later and much of that ‘far fetched’ technology is already here and not much different then what was depicted way back in 2002.

For better or worse, there is little doubt that we are rapidly moving towards a more natural way of interacting with digital solutions via gestural based interfaces (GUI). As an interface developer myself, I see unbelievable potential in this way of interaction. It seems that many companies are also progressing rapidly to be potential leaders in this space. Microsoft has MS Surface and Natal for the XBOX 360 on the horizon. Apple, although they deny it, could have a touch sensitive Netbook by the end of the year to go along with their current entry, a little known device called the iPhone. (Apple has also apparently recently filed many patents to rapidly progress in this space.) But a surprise entry and arguably the current market leader in GUI’s is Nintendo’s Wii and it’s beautiful user experience.

As we move towards gesture-based computing and gaming, creative’s and software developers have to think about completely new interaction models. And as more and more vendors continue to roll out more advanced gesture based devices (anyone search for a book at Indigo or Chapters lately or been to the Royal Ontario Museum?) a complete paradigm shift is on the horizon for the general public. Just as developers will have to think in a new way, so too will the everyday person when doing previously simple things as checking out at the grocery store or taking money out of an ATM.

Human nature drives us to bond with everything around us: we want to feel connected. Even when current user interfaces hit the nail right on the head with regards to usability, the connection to the device running the application can be immediately lost with a simple CNTL-C or a click on the ENTER button. As intuitive and natural as we make our current interfaces (see some amazing examples here), the unnatural way that we interact with them via soon to be archaic devices such as keyboards, mice or game controllers/joysticks holds us back from the much more intimate experiences that GUI’s promise to deliver.

Watching a child playing a video game is a good example that can illustrate our innate yearning for more natural digital interactions. Children find it much easier to suspend their disbelief then adults do. This allows them to feel more connected to a video game for instance, regardless of the graphical complexity. For example, when my daughter plays a driving game with a standard controller, although she has taught herself to use the thumb sticks and buttons to manipulate the digital avatar, she has a natural tendency to turn the controller in the direction that she wants the car to move. We’ve all done this while playing Super Mario Bros. on the NES when making Mario jump with the press of a button usually constituted the gamepad jerking upwards. (In many instances right out of the hands of the user, strictly speaking from experience of course.) Companies have been aware of these tendencies for years but due to hardware limitations have not been able to properly harness this until recently. Nintendo dabbled with mixed results in the eighties with their Power Glove that was based off of NASA technology at the time. It promised to allow players to literally reach into the game and manipulate objects on screen with nothing more then a gesture of the users hand inside the glove. Other companies have released peripherals for gaming systems over the years that allowed players to partake in games based on fly fishing, golf, playing maracas and many other types of experiences. However, it wasn’t until Nintendo released the Wii that gesture based games, and technology for that matter, moved into the mainstream.

Gestures allow us to be more human: they facilitate natural interactions, which provide a more emotional connection between the user and the gestural application. The ability to temper the experience with the size or speed of gesture is also key to driving a much deeper connection then what is currently possible on most of the devices we use today. A gesture-based application should react in different ways, depending how the user is interacting. This is far removed from the current click or don’t click set up we are all used to. It allows users to experiment, and as they do, they are brought closer to what it is they are interacting with. An almost transcendence experience should be possible.

Gesture applications also allow us to be much more social. Nintendo has captured this perfectly with the game Wii Sports that is bundled together with every Wii video game system. Because users don’t have to learn complicated control schematics to pick up and play the games contained in Wii Sports: they feel more inclined to interact in a social setting. Brands will be able to capitalize on this movement by making a normally solo experience – such as browsing the latest and coolest online Flash campaign – into an experience that you can and must do with friends and family. Brands will also be able to offer much more useful solutions to the consumer over the net or special kiosks set up in sporting goods stores for instance. A great way to build brand loyalty could be to allow users to log on to Nike’s newest tennis gear site, select a racket, and then interact with it the way you would in the real world. Sound farfetched? Believe it or not, this type of technology already exists and promises to be a part of the mainstream public within the next 12 months.

Project Natal Demo

Arguably the biggest player entering the GUI space is Microsoft, who revealed to the world at the latest Electronic Entertainment Expo (E3) in Los Angles this year, a new add on device to their popular XBOX 360 video game system currently named Project Natal. The demonstration that followed their announcement during the press briefing, which was introduced by Steven Spielberg himself, was to say the least, jaw dropping. The demo was the closest thing to what was seen in Minority Report then any other real world consumer technology that has come before it. What sets it above and beyond other similar devices is the total absence of a controller of any sorts. The user themselves become the controller. A sway of a hand to the left could cause a tower of interactive blocks to tumble over. Or kicking a virtual soccer ball just as you would in real life is a reality Natal promises to deliver. Interaction without the use of a controller is actually beyond what was conveyed in Minority Report, where Tom Cruise in that movie still needed a glove to interact with the super information system. During further demonstrations of Natal in action, users could scan in real world objects such as skateboards for instance and Natal would recognize this and implement a digital version that could be interacted with, all in real-time. Famed UK game designer Peter Molyneux also demonstrated an interactive avatar named Milo that could recognize users facial expressions and respond accordingly to them. This particular demo was a tad unnerving because of how well it worked. I can see at one end of the spectrum a generation of children isolating themselves with realistic virtual avatars that seem to understand them better then their real world friends or family for that matter. At the other end, however, the unbelievable potential for learning interactions cannot be overlooked. Imagine software that could enhance education where students could have their assignments checked by the avatar by simply holding up the written copy to the screen. Or imagine learning to dance the salsa with a virtual dance instructor. Current advances in home 3D technology coming via companies such as Next 3D, Real D and others, promise to make these experiences even more immersive. If harnessed correctly, Natal and other similar technologies have the potential to completely revolutionize and breath new life into a multitude of industries and mainstay institutions such as education.

As a culture, we should embrace these potential new ways of interacting and thinking. Many who may be afraid of computers (yours truly if I’m not computing or developing on a Mac) may find interaction with GUI’s to be more intuitive and natural then today’s experiences. New ideas from previously unattainable demographics may finally have a voice. New concepts in teaching and learning interaction will be possible. The potential is limitless and that has to be the most exciting thing since, well since Spielberg gave of a glimpse of the future we’re now living in.