The Art and Science of Games

By Michael Dorsey

Robert Lindeman demonstrates the TactaVest, which uses modified pager motors to deliver the sensation of touch to explorers of virtual environments.

Video games, which generated nearly $12.5 billion in retail sales in the United States in 2006, constitute one of the fastest-growing sectors of the entertainment industry. Despite their contribution to economic growth and technological advancement, electronic games, until recently, have been judged too frivolous a subject for academic study.

"People who study film had the same problem 50 years ago," notes Mark Claypool, associate professor of computer science at WPI. "But in the past five to 10 years, games have become a first-class research discipline, even if there aren't a lot of people working in the field yet."

Claypool is co-director of WPI's Interactive Media and Game Development program, one of the nation's first bachelor's degree programs in the field. Reflecting the philosophy that the best games are a perfect marriage between technology, art, music, and story, WPI's was the first such program to equally emphasize the technical and artistic aspects of game development. To that end, its faculty is drawn equally from the Computer Science and Humanities and Arts departments.

In addition to preparing tomorrow's game developers, these faculty, through their research and scholarship, are advancing the art and science behind video games. As the following vignettes demonstrate, much of that work focuses on making games truly immersive artificial worlds that draw players in and keep them engaged.

Beyond Sight and Sound

You're fighting virtual armies of ogres to reach hidden treasure. Suddenly you become aware that a monster is nearby. You can't see or hear it—but its stench is unmistakable.

Robert Lindeman, assistant professor of computer science, wants to enhance the game experience by letting players not only see and hear artificial worlds, but touch, taste, and smell them.

His first target was touch. He has developed virtual reality technology that uses modified pager motors to deliver carefully timed and coordinated vibrations to the skin. The motors are integrated into a garment, the TactaVest, which can simulate the sensation of bumping into a wall or being struck by a sword.

Lindeman is also working on smell. Imagine a restaurant website complete with the aroma of the entrées, he muses. With researchers at the Advanced Telecommunications Research Institute International in Kyoto, Japan, where he has spent the past four summers as a visiting researcher, Lindeman is exploring ways to do just that. One of the most promising is an air cannon that shoots small smoke rings.


Rosenstock brings a sense of play to his custom-built electronic instruments, like this video DJ, which mixes and combines snippets of historical film footage.

You're at a gallery, chatting with guests, when you notice that a soft sculpture is talking to you—in your own voice. It's feeding back snippets of your conversation in a sort of musical composition.

The sculpture, included in PURE, a 2006 interactive art show in Brighton, Mass., is the creation of Joshua Rosenstock, assistant professor of humanities. An artist and musician whose work sits at the intersection of video, sound, and interactive media, Rosenstock sees numerous connections between his art and electronic games.

He often borrows game technology to create unique interfaces for his works—in an installation, for example, that permits participants to navigate through a video production using a joystick. In addition, he says, play is a central theme of his work, something that is especially apparent in the electronic instruments he has invented.

They include a video DJ, which enables a performer to improvise with video clips much as a music DJ mixes and combines fragments of musical recordings. Snippets of historical video can be continually remixed into audio and visual collages, forcing the viewer to focus intently on the footage and draw new meaning from it.

The sonic sculptures included in the PURE exhibit are the beginning of what Rosenstock hopes will be a flock of autonomous robots that will interact with viewers by sampling their conversations and playing them back in a coordinated but spontaneous chorus. "You can think of them as thieves that steal your sounds," he explains. Depending on how one views it, Rosenstock says, the installation can be a commentary on the growing reality of covert surveillance in our world or a mirror, showing us what it is like to be passive media consumers.

While touch adds a powerful new dimension to games, it may be even more valuable in simulations and other "serious games" that are designed to teach while they entertain.

Ready, Aim, ...Wait

The Waking Dreams

Can games ever become art? Joseph Farbrook, assistant professor of humanities, says it is a real possibility, one that he and other digital artists are working to bring about. Photography and film became art, he notes, when they turned self-reflexive—when they become vehicles for talking about themselves. "We are beginning to see the Internet as a medium for fine art," he says. "Perhaps, in time, games and interactive environments will become the content for art—what the art explores. They may become the next hot art medium—at least I hope so."

In some of his own art, Farbrook creates game modifications and machinima (movies made with game engines) that, he says, "play on the idea of a game being a simulation. I am interested in the relationship between the simulation we are doing unconsciously in a game and the ones we follow unconsciously in our daily lives." (He notes, for example, that money is a simulation of work, though we don't think of it that way.)

The allure of simulations, Farbrook notes, accounts for the popularity of video games and the obsession with ever more powerful game consoles. "They mirror dreams, in a way," he says. "They're lucid dreams, where we control the elements. I think that is what we are looking for in games, a waking dream."

You go online to play Halo 2 against other gamers. Materializing in an alien landscape, you spy an enemy soldier. He swings his weapon toward you, but you fire first, with deadly accuracy. Inexplicably, it is you who falls mortally wounded.

You've become a victim of latency, one of a number of technical shortcomings that can set online game players' teeth on edge. As online gaming grows rapidly in popularity (a 2006 report projects that revenues from online games worldwide will grow from $3.4 billion in 2005 to over $13 billion in 2011), addressing issues like latency will become increasingly important.

Latency is the delay between the time you click a mouse or squeeze the trigger on a controller and the time that the result appears on screen. Many factors influence it, from the time it takes light to travel over fiber-optic wires, to congestion caused by heavy network traffic, to busy game servers, notes Mark Claypool, who has studied the impact of latency on gamers. He says tolerance for latency varies depending on the type of game. First-person shooter games, where success depends upon being able to accurately track and hit targets and where a sense of immersion is especially important, are the most sensitive. Delays beyond about 100 milliseconds can seem almost intolerable.

"Online games are continually struggling with latency," Claypool says. "But even worse is variance in latency. It's when it goes up and down that it drives us nuts. This area hasn't been looked at much, and it appears to be a common problem."

This Is Not a Game

A screen shot from the game Titan Quest, one of the top five games of 2006 according to PC Magazine. Dean O'Donnell, co-director of the IMGD program, wrote dialogue and story elements for a new expansion pack for the game, Titan Quest: Immortal Throne. For every possible encounter between the player and a character, O'Donnell wrote seven separate responses for the character to say. The essential information needed to advance the player's quest had to be revealed in the first line. Subsequent lines added greater detail.

You see an ad for a blood drive sponsored by a high-tech firm and email the company. The reply includes a seemingly random set of letters. Decoded with a cipher, they become the address for a secret website warning of the company's plan to turn WPI students into zombies—unless you act!

You've stumbled into an alternate reality game (ARG), one of the latest and most intriguing trends in immersive gaming. This scenario was developed in 2006 by students in the course Storytelling in Interactive Media and Games taught by Dean O'Donnell, administrator and instructor of drama/theatre and co-director of the IMGD program.

For five days, students who stumbled into the game solved a series of puzzles, yielding clues delivered through emails, phone calls, and interactive messages. "People said it was the most compelling game they had ever played," O'Donnell says. ARGs are also called viral marketing since many (like I Love Bees, which promoted the game Halo 2) are product tie-ins. Their success lies in their ability to draw players into a world of intrigue that seems real. "The idea is that it's not a game," O'Donnell says. "It's really happening."

As with any game, the key to maintaining a sense of immersion is good storytelling. O'Donnell, an accomplished playwright, recently gained insight into the unique challenges of writing for games when he worked on an expansion pack for Titan Quest, in which characters embark on quests inspired by Greek mythology. O'Donnell helped craft the story and wrote nearly 2,000 lines of character dialogue.

"Each time a character is clicked on, he has to say something," he notes. "And you have to account for every logical action the player could take. Every line is like starting from scratch. There is no flow, no ‘he says, she says.' The dialogue assumes that the player has spoken."

Facing the Music

You're exploring a virtual civilization, but something keeps pulling your mind away from the fantasy. You become aware that the game is playing the same musical phrase over, and over, and…

"Music in games is completely underdeveloped," says Frederick Bianchi, professor of music. "As in films, it's always the last thing to get done." Even when game music soars (as it does more and more, with the rise of a dedicated game music industry), it has failed to keep up with technological advances that have propelled other aspects of the game experience.

Typically, game music consists of a collection of themes that are selected and played to match the action. Clever programming can seamlessly segue between themes, creating the illusion that the music is reacting to the actions of the player. In the not-too-distant future, Bianchi says, it literally will.

"Work over the past 20 to 30 years has been leading to algorithmic composition—the ability to create music on the fly," he says. "For example, if the situation calls for music in the style of John Williams, you could program in Williams's musical DNA and have the program produce compositions at will."

Someday, Bianchi says, games may be able to not only respond to what players do, but sense what they're thinking and feeling. "It will be a difficult challenge because you will have to engage cognitive psychologists and other specialists who can tell you how the brain works and how people interface with machines. It is beyond a computer science or a music problem."

People's ability to deal with the output of machines (he calls it human bandwidth) intrigues Bianchi, who has spent more than two decades developing a real-time interactive music performance system, the Virtual Orchestra. The groundbreaking technology, which can simulate the sound and behavior of a live orchestra, has been used in more than 15,000 performances worldwide.

"The human bandwidth applies especially to gaming and interactive media," he says, "but we have a long way to go in capitalizing on it."

Emmanuel Agu studies graphics for mobile computing devices. He has developed applications that can help save mobile devices' precious battery power, render diffraction photo-realistically, and scale graphics automatically so they appear properly on devices ranging from PCs to PDAs.

Games on the Go

You're lining up a putt in an online golf game when you realize you have to run to catch a train. A half hour later, you continue the game on your cell phone, surprised at how closely the small screen's image resembles what you saw on your PC.

Games played on mobile devices like cell phones and PDAs represent one of the biggest growth areas in online gaming. Such devices present a number of challenges to game designers, notes Emmanuel Agu, assistant professor of computer science.

Among the most significant is the fact that the graphics, processing power, and wireless communications seriously tax a mobile device's limited battery power. In addition, making graphics designed for computer monitors look good on tiny displays can be a time-consuming ordeal.

In his research, Agu is tackling both issues. He has developed a tool called Power Spy that monitors every process taking place inside a mobile device and identifies the power hogs. It helps pinpoint inefficiencies in software and hardware. "You have to attack power usage on every level," he says.

Agu is also developing techniques for scaling graphics automatically. Currently, taking a game optimized for a PC and adapting it for a small cell phone screen requires months of painstaking work by programmers. "We're looking for ways to encode the data so we can do the conversion automatically for any platform," he says.

Agu is also working to make graphics more realistic looking, no matter what screen they appear on. In particular, he is developing algorithms that can render diffraction, which causes the rainbow reflections from the surface of a CD-ROM. "It is very difficult because the behavior of the surface changes over the entire spectrum."

Like all of the faculty connected with the IMGD program, Agu is aware that he is breaking new ground and developing solutions that have the potential to affect the direction of a major industry. "Because research in games is a new field," he says, "people working in the area are defining the problems that have to be solved. We are, in a real sense, helping to shape the field."