May 2012

A Brave New World for Augmented Reality

Iulian Radu

Augmented reality (AR) provides an experience where users can see computer-generated imagery and information displayed in real environments. Rather than simulating another world (a la Star Trek’s holodeck), augmented reality enhances your own. One popular method for this is to look through a display (such as on a phone or tablet) at images and sounds in a room or on a street that aren’t physically there but create a new dynamic experience at the spot you are standing.

AR technology has slowly made its way into more and more consumer electronics over the past few years including smartphones, televisions, gaming systems, and tablets. Now Google’s Project Glass - a AR-related concept - is stoking many people’s imaginations and pushing future opportunities for this burgeoning technology field.

Iulian Radu, a Ph.D. student in Human-Centered Computing at Georgia Tech, is interested in understanding how to design augmented-reality experiences that are suitable for young children, 6 years of age and older. In the AR community there are no "best practices" or guidelines for how to design AR experiences that are suitable for children of specific ages. The difficulty is that one AR design for a specific age group may not be suitable for another, because children develop over time and they have different capabilities and limitations due to their physical, cognitive, emotional and social development. Radu discusses his research focus in determining how developmental psychology can help others to understand and improve AR designs for children.

Q. Augmented Reality isn’t widespread like other consumer technologies, and adults, let alone children, may not be familiar with it. What is it about AR that you believe is significant for children as they continue to be exposed to technology at younger ages?

Augmented reality brings digital content into the physical world, and allows children to use their whole body to interact with the experience. I feel that the combination of these factors offers great potential for enriching children’s lives.

Entertainment systems based on natural interaction, such as the Kinect and Wii, are showing us that children are excited about moving their body while playing games, and that the whole-body interface is an intuitive way for new users to become familiar with a system. AR experiences are inherently motivational for children due to the physical nature of the interactions. There is also research in the area of embodied cognition, which tells us that there is a relationship between our bodily experiences and our conceptual understandings of the world. Thus, I feel that AR interfaces can be potentially useful as a new pathway to learning.

Besides allowing users to use physical motions to naturally interact with content, AR allows children to visualize things that are otherwise inaccessible. For instance, it can be used to give physical existence to 3D mathematical functions, or to materialize the magnetic fields around objects. It can also give children a better understanding of space and time – it can replace present-day buildings with their past or future counterparts; it can scale up an atom to the size of a classroom, and scale down the solar system to the palm of a hand. Further, AR can allow children to see abstract concepts as concrete objects, for example seeing a city’s crime rate as dark clouds above the city blocks, or see the contents of a book as a web of interrelated concepts that you can physically manipulate. The applications are fascinating, and the availability of this technology on mobile devices makes it perfect for delivering on-demand educational experiences.

There are many challenges to harnessing the power of this technology for benefitting child users. A great hurdle is that there are few examples of AR systems for children, thus there are no established guidelines on which AR designs work for which age groups. My research explores the question of how to design AR applications such that they are usable by young children.

Q. Your current research focus is using developmental psychology to guide augmented-reality design for children. What does this entail and what have been some of the results of the work? 

There are incredible developments that occur in the first 10 years of life. A newborn human organism transforms into an individual that fluently communicates and empathizes with others, navigates and manipulates its environment, analyzes situations and uses abstract thinking to achieve its goals, masters the use of tools and artistic instruments, and learns other abilities that late-childhood children are skilled at. If we are to design augmented-reality technology for children, we need to understand the characteristics of children at different ages, and we need to understand what kinds of AR designs are suitable and problematic for these children.

My research is focused on this topic, and my approach has been to follow a three-part process. One part of my work is to understand children’s capabilities and limitations at different ages. For me this comes from studying developmental psychology literature, collaborating with psychologists from Georgia Tech and other local universities, and volunteering at a local elementary school. From these activities I have identified a set of children’s abilities that could be necessary for interacting with AR applications, yet which may be undeveloped in children 6-9 years old – for instance, the ability to pay close attention to two spaces at the same time, such as when performing precise two-handed manipulation on a tabletop while looking at an AR view on a monitor; or, the ability to imagine what an augmented physical space looks like from another person’s perspective, such as when planning actions in a 3D multiplayer game.

Another part of my work involves designing various kinds of AR applications for children. In building and evaluating these systems, I am exploring children’s reactions to AR, and understanding how developmental psychology can be applied to AR design. Most of my systems have been exploratory, but my approach now is to use developmental psychology to design games of increasing difficulty in order to determine exactly which AR designs are problematic for children of different ages.

Finally, a third part of this research involves refining a list of usability guidelines for designers of children’s AR applications. By understanding which AR designs are problematic, and which psychological and physiological factors underlie those issues, we can generate guidelines that designers can use to create effective AR applications and/or to explain children’s reactions to AR applications.

Q. AR user experiences can vary significantly given that the technology is relatively young. In developing AR design guidelines, what are some factors you’ve discovered that may be needed in the design process for children?

In the spirit of user-centered design, I believe that understanding users should be a central activity in the process of designing technology and evaluating its impact. In the domain of applications for children, it is valuable to have knowledge about the psychological and physiological characteristics of children at different ages. When considering AR usability specifically, I believe it is important to acknowledge children’s developmental characteristics in the domains of motor skills, attention, spatial awareness, and logical thinking, because these will impact children’s experience of most AR applications.

However, children live in subjectively different worlds than adults, and I also believe that no matter how much knowledge we have about children’s abilities, they will always manage to surprise us by interacting in unexpected ways with our applications, or show us new ways of looking at the experiences we build.

Because these subjective differences exist, and because there is a lack of knowledge about child users in the AR community, I have learned that it is important to involve children early in the design process. I design AR experiences and expose them to children play-testers, expecting to go back to the drawing board in response to children’s reactions. Besides just observing as children play with my software, I treat children as informants, talking to them about their experiences, asking for improvements, and I have even used arts-and-crafts to understand children’s visions of different uses of the software. I’ve found these activities to be invaluable, often identifying useful features to add, and giving me a better understanding of children’s experiences in AR.

I should also mention that I am a partial believer in the “users do not always know best” philosophy, and I believe that when developing applications for children it is valuable to involve the expertise of other stakeholders, such as teachers and parents.

Q. You’ve gone beyond theoretical design and developed your own AR technology that has been made available for educators and others. Can you give details on these systems and what you learned from them?

Developing AR systems for children is a fun and rewarding experience for me, because I get to literally play with my software as I develop it, and because exposing it to children shows me how technology can spark wonder and enjoyment.

I’ve developed several AR systems over the years. The first was AR Spot, a webcam-based augmented-reality authoring environment based on MIT’s popular Scratch programming platform. In developing this, I thought about how older children (10-13 years old) would think about programming augmented-reality games. Developing and evaluating this system has taught me the importance of considering spatial abilities in AR – children may have difficulty with 3D visualizations, and working with 3D frames of reference; thus, an AR programming environment may avoid or scaffold this kind of spatial thinking. It also taught me that children intuitively apply real-world knowledge to interacting with AR – they expect virtual objects to obey physical laws, even if the virtual objects look two-dimensional; thus AR designers should support, or at least account for, these expectations.

The second set of systems I’ve developed were two handheld games, Spintopia and PuppyPlus. These were developed by collaborating with other students from Georgia Tech and SCAD (Savannah College of Art and Design), and were intended to explore children’s use of handheld AR. Spintopia is an art-making game, in which children draw 3D shapes through their mobile device. PuppyPlus is a game for teaching elementary-school mathematics, and is played by using a handheld phone and several physical props. In evaluating these, we found that younger children (6-8 years old) had difficulties with physical manipulation of the AR experiences – they had trouble performing precise movements, and using two hands to interact with the systems. We also found attention issues related to the technology – children were so focused on the game effects that they did not notice when the AR tracking would fail to work.

Now I am working on a second set of games, to be played by young children (6-9 years old) with mobile phones and tablet devices. These games are designed to educate children about animal habitats and about the human immune system, and their design has been informed by developmental psychology. Through these, I am further exploring how different AR designs impact children’s spatial cognition and physical interactions. These games were tested at the 2nd Annual U.S. Science and Engineering Festival in Washington D.C. in April. We had more than 4,500 visitors come through the booth so the games got quite a trial run.

Q. What application areas might benefit most from your current research and what might a deployment of AR in those areas look like?

In my research, I am mostly focused on psychological factors, and this makes me feel that augmented reality will have fascinating applications in educational and therapeutic domains. Especially with the proliferation of AR glasses, this technology has the ability to radically transform the user’s perception of their surroundings, and, in the process, may permanently alter the user’s relationship with his or her world.

In the future, we may be able to create applications with the goal of enhancing wonder and creativity, by virtually reshaping all the buildings around a user’s neighborhood, or allowing the user to artistically express his or herself by painting on buildings, people, and even on the sky. Or, we could create applications that don’t enhance, but instead simplify reality, for instance by reducing the colors of the world, or transforming the world into a cartoon; if done well, this application may be beneficial for some individuals on the Autism Spectrum that are overwhelmed by details of facial expressions. Another fascinating application of AR is to give physical representation to our abstract ideas, allowing us to see and manipulate concepts as if they are physical objects, thus giving us a new way of communicating and analyzing our thoughts. I feel AR has much potential to alter our perception of reality, through such applications and many more that may become possible in the not-so-far future.

However, as it stands now, this technology is still in its early stages, and we don’t fully understand where its most significant uses are. I believe that as the authoring tools become easier to use, as we develop best-practices for designing with this technology, and as more non-technical passionate individuals become involved in creating AR experiences, the full potential of this fantastic technology will become realized.

Links:

Iulian Radu homepage:
http://www.cc.gatech.edu/~iulian/

AR Spot authoring environment:
http://ael.gatech.edu/lab/research/arspot/

Puppy Plus:
http://www.youtube.com/watch?v=ArUp1gxUrOU

Spintopia:
http://www.youtube.com/watch?v=0poxJKNVLIM

Argon games (shown at the U.S. Science and Engineering Festival):
http://argon.gatech.edu/games

Researchers are studying the explosion of social media activity as a way of gauging "certainty" in an age of Internet news. A team led by Mengdie Hu wanted to distinguish rumor or "uncertain" tweets made that night from those that were "certain" regarding bin Laden's death.

Last year's biggest news story - the death of Osama bin Laden - is now a key to understanding just how deeply Twitter has affected the news universe. The Georgia Institute of Technology will release a study today about coverage of bin Laden's killing that may be the most comprehensive yet in showing how news spreads on Twitter. Researchers at Georgia Tech worked with researchers at Microsoft Research Asia and University of California-Davis to analyze more than 600,000 tweets sent in a two-hour period, stretching from minutes before the first rumor of Osama bin Laden's death to tweets surrounding confirmation of his killing by U.S. forces. Among the key findings: The majority of people reading the early tweets believed they were true, even before they were confirmed by mainstream media, and celebrities played a key role in disseminating the news... "Rumors spreading on Twitter is one thing," said Mengdie Hu, a Ph.D. candidate in Georgia Tech's School of Interactive Computing, who led the study. "Determining if they are true is another, especially in this era of social media and the rush to break news."

Faculty and staff from Georgia Tech will be showcasing the Institute’s cutting-edge research to thousands of children through interactive and entertaining activities. At Georgia Tech’s expo booth, the activities include augmented reality, nanotechnology, and lasers.

When Things Fall Apart, an experimental campaign launching Friday, deconstructs and reconstructs your Twitter avatar over three days when you donate $10 or more to the Red Cross. Using a visual language called Processing, your picture will be scattered and periodically rebuilt over 60 hours in 12-hour increments. When Things Fall Apart thematically mirrors the work the Red Cross does, rebuilding communities that have been shattered to pieces by natural disasters. The campaign fits into Georgia Tech Professor Eric Gilbert’s research on Twitter mobilization through changing profile pictures in both organic and orchestrated campaigns. Since people adopted green icons during the Iranian uprisings in 2009, cause campaigns have used profile pictures to generate awareness. "I’m pretty fascinated by these campaigns, but they’re also really easy to fake, at max requiring a very basic knowledge of Photoshop," Gilbert told Mashable. "My hope was that if we inject serious computation into this can we get people to pay money and send it to a worthy cause."

The last revolutionary moment in multiplayer gaming was when it went online. Now asynchronous gaming lets you and a friend compete on your own schedules. Ian Bogost (Digital Media) talks to Wired about this next gaming revolution.

Google's Project Glass eyewear caused quite a stir when the company unveiled it this week. But now augmented reality experts, including Blair MacIntyre (Interactive Computing) have cast doubts on whether Google could pull off what's shown in the promo video with the hardware.

Google has sparked an online tizzy with its Project Glass video, a breezy, aspirational clip depicting life after we all sport the search giant’s snazzy new designer spectacles that put the digital world at our, er, nose-tips, from the moment we rise. Bloggers such as Blair MacIntyre, director of the Augmented Reality Lab at the Georgia Institute of Technology in Atlanta, ask the obvious questions, such as "is it a good idea for Google to hype expectations about a product that it cannot possibly deliver?" The future product’s technology builds on many existing smart functions, such as location-based technology and targeted advertising. But, points out Mr. MacIntyre, many of those technologies have also fallen short of expectations. "I was an early adopter with those apps," he says, but notes that after fumbling through notification after notification of only mildly useful information flowing incessantly into his smart phone, "I stopped looking at each one as it came in."

Jenay Beer with the PR2 Robot

Robots, once the domain of futuristic imaginings and fiction, are steadily becoming a reality in many sectors of modern society, including manufacturing, defense and healthcare.

A Georgia Tech research team is now imagining a different category of robot, one that would allow aging populations to live independently into their golden years with the aid of domesticated robot helpers.

The PR2 project is a multidisciplinary effort through two Georgia Tech research labs - the Human Factors and Aging Laboratory (Wendy Rogers) and the Healthcare Robotics Laboratory (Charlie Kemp) – focused on the development of assistive capabilities for the Willow Garage PR2 robot. The emphasis is to develop the life-sized robot’s functionality so that it can assist people in tasks that allow them to “age in place,” which refers to older adults sustaining independence in their own homes.

Jenay Beer, a Ph.D. candidate in Engineering Psychology and member of the PR2 team, is focused on identifying the needs of senior citizens that robots can assist with and how that would affect the design implementation of assistive robots.

Despite a lack of robots on retail shelves and older generations that grew up without computers, the research team has found that older adults would eagerly enlist robot help.

“If the benefit of using a robot is clear, then older adults are willing to use home-based robots,” Beer says. “But that’s the key – the robots have to assist them in some useful way.”

Akin to hearing aids, robots are an assistive technology that can improve the quality of life for individuals. Tasks that might consume too much energy or time, or both, are ideal for robots to tackle, according to the study. Robots can also perform the tasks that older adults may not enjoy doing such as housecleaning.

Participants, ages 65 to 93, viewed a video of the PR2 robot in action, followed by a written survey and structured group interview. The study results showed that robots were preferred over human helpers in 28 of the 48 identified home-based tasks.

Cleaning kitchens, bathrooms and windows were all jobs willingly handed off to robots. Respondents were less preferential toward their artificial assistants when it came to sorting mail, laundry and washing dishes - the latter presumably because the robots might stop working if they became wet. For organizing and fetching tasks, respondents preferred robots when it was necessary to reach in high or low places, retrieve items or pick up heavy objects.

The preferences were only for certain tasks, Beer says, but they show the potential for assistive robots in the home. The next step is to understand in more depth why older adults hold certain preferences.

Beer presented recommendations for robot design at the 7th Annual ACM/IEEE International Conference on Human Robot Interaction last month. Beer co-authored the paper, “The Domesticated Robot: Design Guidelines for Assisting Older Adults to Age in Place,” along with PR2 team members Cory-Ann Smarr, Tiffany Chen, Akanksha Prakash, Tracy Mitzner, Charles Kemp and Wendy Rogers.

Recommendations from the team include making robots customizable and the ability for them to collaborate with users.

“Some older adults voiced a concern of becoming ‘over-dependent’ on the robot,” Beer says. “They suggested that a robot might do a difficult aspect of a task, while they do the rest. The human and robot would be working together.”

The transformative effects of robots in older adults' homes are potentially limitless. A robot that is always on and alert could prevent falls and other accidents. Also, adults looking after young children as well as aging parents – the so-called sandwich generation – could have a new piece of mind with robot helpers.

“Robots will become ubiquitous in the home only if we can understand the users’ needs,” Beer says. “We have so much yet to understand how and why older adults might accept robot assistance. Concepts of long-term acceptance, when to introduce robots, or changes in acceptance over time have yet to be addressed.”

Beer will talk about the future of robotics and her role in Georgia Tech’s endeavors within the area as part of TEDxGeorgiaTech 2012, taking place April 7 at the Academy of Medicine in Midtown Atlanta.

Writer: Josh Preston (jpreston@cc.gatech.edu)

Photo: Jerome Choo