As seen on the EA Capture Facebook page.

FaceForward

Battlefield 3 is different. We built the facial animation pipeline used for the interrogation scenes and I’m rather proud of what we accomplished. EA Capture was small studio with an aggressive timeline to create a new technology for one of our company’s biggest titles. And we had to do this while maintaining the same workload and quality for our other game teams. It was an incredible experience and if I had to do it again… I would do many things differently. Though, that shows how hard we were pushing our limits and how much better this technology will become. It’s next generation technology used for the current generation of games.

I’m finally at the point where I can watch the cutscenes and see how good they look, instead of seeing the things I want to change. More than anything, I’m excited about what our creative talent will do with this in the future.

As much as I want to celebrate the achievement of my team, I am in awe of the wizards at DICE who actually got this stuff working in Frostbite. You can see the fruits of their labour below.

If you can’t view it on YouTube, check it out on GameTrailers.

Trailer:

Despite his ailing health, the news of Steve’s death was not something that I expected.  The man who conquered the world with functional elegance is gone and my deepest sympathies go out to the loved ones he left behind.

In his 2005 Stanford commencement address, he left us with several musings on life, death and happiness.

“If today were the last day of my life would I want to do what I’m about to do today?”

“Remembering that I’ll be dead soon is the most important tool I’ve ever encountered to help me make the big choices in life.”

“Remembering that you are going to die is the best way I know to avoid the trap of thinking that you have something to lose.”

“Your time is limited so don’t waste it living someone else’s life.  Don’t be trapped by dogma, which is living with the results of other people’s thinking.  Don’t let the noise of others’ opinions drown out your own inner voice.  And most important, have the courage to follow your heart and intuition.  They somehow already know what you truly want to become.  Everything else is secondary.”

RIP Steve Jobs

I have been very busy since my last post.  In addition to my own projects, I found myself employed.

In June 2010 I was contacted by Electronic Arts for an interview.  By the end of the month, I had moved from Hamilton, Ontario to Vancouver, British Columbia and was working as a Software Engineer on FIFA Online.  In September, I toured the EA World Wide Motion Capture Studio and, within a few weeks, was working for them doing research and development.

The following is the first demo of a project that I was a part of.  It demonstrates a new facial motion capture rig that we developed.  This technology allows us to create compelling performances in which the performer does not need to look like the model.  Essentially, I could do the performance and the coach would act it out.  If I furrow my brow, he will furrow his.  You will see every wrinkle, skin tone change, etc.

The coach is played by a local coach. The team are my multi-talented co-workers (I make an appearance at 0:53 – #72). As you can see from the texture/model recycling, we are a pretty small group.

This was originally posted on the EA Motion Capture Studios Facebook group. Watch it on YouTube in 720P.

Perimeter Institute for Theoretical Physics

I just received my University of Waterloo alumni newsletter…

Hawking is here!  Hawking is here!

[also posted on GameCareerGuide.com]

Do games tell stories?

Sure, text, artwork, voice acting and cut-scenes can all arguably tell or help tell a story, but how can you truly say that the game itself is telling the story? And by the game, I mean the actual system, the units and rules that create the possibility for gameplay. Is gameplay a form of storytelling? Maybe not in most games (to avoid the argument), but if we wanted to conceptualize gameplay as storytelling, how would we do it? And if we wanted to make a game that told its story well, what would it take?

In short, and I’ll go into more detail later in this article, yes: it can be useful to think of gameplay as a medium through which players experience a unique form of storytelling. Maybe you’ve experienced it yourself where for one brief moment everything—the characters, the sounds, the visuals and what you were doing—all seemed to click, and you felt truly engaged in the story being told. It’s something that many gamers have felt at some point, but that no one has yet been able to consistently reproduce. “It” eludes us not because we lack the tools to describe or evaluate it, but because it crosses so many fields and disciplines. Theories of fun and swords and circuitry, research into expressive AI and dreams of Hamlet on the Holodeck all bring us closer to understanding it, but none provide that one true holistic vantage point from which a game designer can envision how to truly tell stories well through gameplay.

A holistic approach to storytelling in games has to consider many literary and filmic concepts like story, plot, character development, cinematography, lighting, audiography and “editing”. But unlike film, a holistic approach must also consider the game mechanics and expressive processes that determine the above (no small feat), all the while recognizing that games are interactive, and have spatial and haptic dimensions. Is it any wonder that a holistic view of storytelling in games has eluded us for so long? The solution isn’t to mash the concepts together and hope for the best. Putting Steven Spielberg, Conrad Hall, Syd Field, Jorge Luis Borges, Chris Crawford, Nobuo Uematsu, and Michael Mateas into a room probably won’t produce anything worthwhile because they have no common framework on which to have a meaningful discussion.

To find that common framework, we have to go up the conceptual tree to find what all of these seemingly disparate disciplines share. And that shared concept is communication. Ultimately, they are all means of getting an idea from person A, across some medium, to person B. But that net might be cast a little too wide for our purposes. Storytelling is a specific form of communication, a form studied for thousands of years by that often misunderstood field of study: narratology (as it’s called today). But I’d like to ignore Aristotle for once and instead shed some light on the modern founders of narratology, the Russian Formalists, who a hundred years ago decided to analyze literature as if the stories it told were complex machines intentionally and purposefully constructed using “devices” or “functions” that serve particular purposes. It’s from this concept that we get the term “plot device”. This approach to understanding storytelling is interesting because today we use complex machines to intentionally and purposefully design and program functions that serve particular purposes in order to tell stories. Somehow, this conceptual similarity has rarely been noticed.

Unfortunately, after years of largely pointless “Ludology vs. Narratology” debates, narratology is seen as a dead horse whose body is periodically dragged out by articles like this one for yet another beating. Except that this article isn’t about using narratology to “understand” games, it’s about giving designers a framework on which they can use all of the tools in their toolkit, not just a few. Narratology is the foundation for a common framework that we can all use to set up and guide the shape and direction of ours stories; game design, cinematography, level design, artificial intelligence, art, sound design, etc., are the tools we use to create a story; and gameplay is the way we as players experience that story. So what is this common framework?

Laying the Foundation

Before describing the framework, we need to build a foundation of commonly accepted terms and definitions to stand on. One of the major stumbling blocks when it comes to talking about story in games is that we can’t agree on what “story” even means. It seems like everyone in this industry has their own definition of story, plot and narrative; it’s again no wonder that no one can agree on anything.

Meanwhile, narratologists has generally agreed on specific definitions of “story” and “plot” for about a century. Although “narrative” has some academic wrinkles left in it to iron out, for our purposes a (fuzzy[1]) definition is easy enough to come to. The basic theory goes like this: a narrative is a linear sequence of events through time where it’s said that things cause things to happen to other things. I think that’s generic enough. Whether we’re reading a book, watching a movie or playing a game, the way we experience reading, watching and playing is as a narrative. Games are experienced as narratives.

But that’s not really the useful part of the theory. What’s useful is the implication that whenever something is narrated to you—whenever you have a narrative—that narrative can be described as being composed of two simultaneous planes, like two sides of a coin: the “content plane” (story) and the “expression plane” (plot). The story is the abstract chronology of events and characters behind any narrative. That movie you want to see and spoiler you avoid reading refer to the same content, the same story, even if they tell it differently.

Plot, on the other hand, isn’t used here in the everyday sense of rising action, climax and resolution; the story arc; and all that. In this article, I’d like to take use a different definition of plot. To narratologists, plot is the order of events as they’re told, and plot devices are used to deliberately create certain effects, express certain meanings. The film Memento (2000, Summit Entertainment) starts at the end of its story, and scene by scene takes the viewer back to the beginning. The difference in Memento between “story order” and “the order that the story is told” is the difference between story and plot. Memento’s reverse chronology is a plot device used to great effect. Without the reversal, the film would arguably be far less effective in engaging the viewer and in expressing that sense of piecing together lost memories.

I hope the above helps clarify how we’ll be using the terms story, plot and narrative in this article. I’m not so naïve to think that I can change the way people talk about storytelling with an article, but I do hope that the basic concept that we can differentiate between what is being told (the story), and the telling itself (the plot) is now apparent.

The point? By using these definitions for story, plot and narrative we can deduce that since gameplay is experienced as a narrative, then gameplay can also be described as expressing a story by means of a dynamic plot. This deduction becomes important when considering famed literary theorist Roland Barthes’s argument[2] that every aspect of how a story is told can be usefully described as a plot device. The choice of words, the sequence of shots, the musical score and/or the visuals that express the story are all meaningful, whether that meaning is intended or not. For game designers, this suggests that every game mechanic, art asset, animation, environment, sound effect, musical score and haptic sensation is made meaningful through play. It isn’t enough to consider how a tweak in game mechanics or the placement of a door will affect the game’s playability; how will it affect the story?

Experiencing Story through Play

As mentioned above, it’s useful to describe gameplay in terms of story and plot. Not all games should be described this way, since a game for gaming’s sake has arguably no reason to worry about storytelling. But with the release of titles like Uncharted 2 (Naughty Dog, 2009) and Heavy Rain (Quantic Dream, 2010), it’s easy to see that good storytelling continues to be an important selling feature.

For games that actually do care about telling a story well, it can be useful to describe gameplay as the expression of a game’s story. In other words, gameplay can be usefully described as a system of both play and plot. Games dynamically produce plot as an emergent experience. When a player plays a game, their agency allows them to experience a different narrative with every playthrough, but importantly, not a different story. It isn’t the story that changes, it’s the plot that does. The story stays the same because, in games, the story is dictated by the game mechanics, art assets, animations, environments, sound effects, musical score and haptic sensations that make up the game. A game’s story materializes itself experientially through the interaction of its many parts. Worries over linear and non-linear storytelling are irrelevant. Gameplay is pseudo-linear at best, but describing stories in terms of linearity doesn’t bring us any closer to telling those stories better. Instead, we should consider that when a game is played, its plot—the expression of its many parts and possibilities—is emergent, dynamic, and never the same twice, and that because the parts and possibilities stay the same, the story stays the same even if the narrative is different. I’m sure that someone could make a polymorphic game where this isn’t the case, or that we could get hung up over stories with multiple, branching storylines, but let’s not pull hairs. Heavy Rain may have multiple endings, and The Sims (Electronic Arts) may have no true ending, but both can usefully be described as telling a story.

In fact, describing certain games this way can be insightful. For example, The Sims is considered a powerful example of a game that lets players author their own stories; except, that’s largely an illusion. In actuality, what players create is a version of a specific kind of story: a story about suburban life, friends, love, marriage, getting a job, having a child, peeing yourself in public, etc. The game mechanics of The Sims provide the building blocks necessary for a player to bring about their own rendition of this story, but they’re limited by the game’s space of possibilities. They can’t create a story about a Sim giving up his meaningless, commercialistic life, moving to India, joining an obscure religious sect and living out his dream of an ascetic life … until one day Carla (from back home) finds him and begs him to please! please come home! The reason players can’t author this particular story is because the building blocks that The Sims provides the player don’t include these potential story events. The story of The Sims is pre-defined by its game mechanics, art assets, animations, environments, sound effects, and musical score, and what players do when playing The Sims is bring about one of the virtually infinite (yet pre-defined) dynamic, emergent plots afforded to them through their interactions with the game’s space of possibilities.

But What about Characters?

I think narratology has an interesting and, importantly, useful way of looking at characters in stories. In narratology, characters in a story aren’t living beings that exist in a fictional world. They don’t think, they don’t have emotions, they don’t feel. Instead, they’re described in a purely functional way. Characters only exist because an author wants a certain action or event in their plot to take place. It’s a subtle change in perspective that constantly acknowledges the fictionality of the story, or in other words, that the way a story is told is a construction and not a projection of some kind of reality. To a narratologist, in Spiderman’s origin story, Uncle Ben doesn’t die at the hands of some burglar because one rolled lawful good and the other chaotic evil, or because the burglar had a higher agility score, or because the burglar had a bad day, really needed some cash and got careless. The reason Uncle Ben dies at the hands of some burglar is to provide the realistic motivation needed by the plot to satisfy the necessary causes and effects that lead to Peter Parker’s transformation into the friendly neighbourhood Spiderman we all know. Peter doesn’t decide to fight for justice because of the guilt he feels over Ben’s death; Peter doesn’t decide or feel at all. Peter, Ben and the burglar are just ink on a page or moving images on a screen, they don’t exist. As far as Spiderman’s origin story goes, the only reason why the characters of Ben Parker and the burglar exist in the plot is to provide a realistic reason for Peter Parker’s transformation from selfish mercenary to altruistic hero.

What does that have to do with game design? Well, what’s interesting is that game designers already see characters in a similar way. When designing gameplay, characters are just collections of functionality encased in mesh or sprite. Narratologists analyze what a character’s function in the plot is, and game designers determine what a character’s function during play will be. And this is where the two meet: once you know exactly what a character’s purpose in the plot will be you can start thinking about what game mechanics  best serve to fulfill that purpose. Or considered differently, when designing character AI don’t fret over simulating what they’re thinking or feeling, instead focus on creating behaviours that act as coherent plot devices that reinforce the story in meaningful ways.

Unfortunately, creating meaningful character behaviours is no simple task. Characters in many games are based on functional roles: hero, quest giver, villain, helper. These are the kinds of roles that narratologist Vladimir Propp discovered were pervasive in fairytales. When we find these character archetypes in games, they do indeed align their game function with their story function, but like most fairytale characters they’re also flat and boring. The game industry has been trying hard to move away from flat characters, but in most cases we don’t know how to approach creating that depth. Back story and cinematics can only go so far in establishing characters. What matters more is how those characters behave during gameplay. If their only role during gameplay is to be a mindless “helper,” then even the most masterfully rendered cut-scene will fail to convince a player that they’re anything but a flat character once the cut-scene ends. The key lies in creating in-game character behaviours that help reinforce their characterization and the story’s themes, and dynamically create moments for the player to experience the story you’re trying to tell.

Agency

So we’ve covered story, plot and characters, but there’s one more element to stories in games that all other forms of storytelling don’t have to worry about: the meddlesome player.

Player agency can bothers us when we think about it too hard. Giving some chaotic player free-reign in our exquisitely crafted world is exhilarating, but having to worry about how they might go against our intentions can be frustrating. Throw in a complex story and what you have in front of you is so sublimely complex that it just seems better to avoid thinking about both the story and gameplay at the same time. How can you tell a genuine story when the player can do whatever they want?

Yet, this way of looking at player agency ignores one basic fact: a player can’t do whatever they want; they can only do precisely what you let them do. If a player can take a game’s mechanics and subvert its story (bugs and exploits aside) then those mechanics weren’t well designed. This isn’t meant to be an insult, just a statement of fact. If every game mechanic is designed from the start to tell a particular story, then no matter what the player does within the limits of those game mechanics, the result should be the story that the game was intentionally and purposefully built to tell. Let’s take a step back and look at what this actually means in practice.

Let’s say an intrepid designer wants to tell a story about freedom, levity and reckless youth. She decides that a good way to tell that kind of a story is through a set of core game mechanics that revolve around flying. Flying: freedom, levity, recklessness—it makes sense. Unfortunately, she realizes during some initial design sessions that a player could undermine her theme of reckless youth by repeatedly smashing themselves into cliff faces and dying. The point of reckless youth is that you feel invincible; such an obvious vulnerability hinders that feeling. What should she do? Remove those dangerous cliffs entirely? Add artificial boundaries that bounce the player away? Insert an all too convenient excuse into the story? These design/story choices may solve the problem mechanically, but they would only serve to undermine or at least dilute the sense of freedom and recklessness she was trying to express in the first place.

Instead, her solution is to change how wall collisions are handled. When the player hits a wall, he will acrobatically bounce off, or by holding down the grab button, just latch on. Now that feeling of reckless invincibility is palpable. Even if a subversive player wants to ruin the story by smashing themselves into walls repeatedly, this mechanic acts as a plot device that ensures the game stays on message by taking “bad” player behaviour and making it nonetheless tell the story the way our designer wants it to be told. Rather than try to create a realistic simulation that panders to the flawed notion that players need absolute, realistic control over a system in order to feel immersed, our designer sees that the story’s needs outweigh the player’s need to feel they have the freedom to commit suicide on cliff faces. Immersion is important, but it’s also important to recognize the degree of simulation needed to preserve that sense of immersion.

These collision mechanics also avoid punishing the player as they learn to play. Not only does it help reinforce one of the story’s themes, on its own it’s also a sound design decision. Maybe near the end of the game our designer will remove this mechanic (accompanied by a credible story pretence, like a broken leg) because, after all, the invincibility of reckless youth is only an illusion…

The key here is that she had to face the fact that either (a) her game’s mechanics didn’t suit the story, or else (b) her story didn’t suit the mechanics. This may be a broad generalization, but I hope that it sparks a shift in perspective: if you’re telling your story in a way that ruins the story, then you’re not doing it right.

Conflicts and Coherence in Dynamic Plot

We’ve all played those games where the game’s story is trying to be serious, but the game mechanics make it ridiculous, or where we know what the story is trying to do, but we’re just not feeling it. A lack of coherence between story and plot, or story and gameplay, unless done for satirical effect, is the mark of bad storytelling. Let’s take a look at a game-related example. In Grand Theft Auto IV (Rockstar Games), the player can go on dates outside of the main storyline with four women: Kate, Alex, Kiki and Carmen. These side quests aren’t just there to be amusing; completing multiple successful dates with these characters will reward the player with tangible, in-game abilities (e.g.: calling Kiki will take three stars off your wanted level). All of them, that is, except for Kate. No matter how many successful dates you go on with Kate there is no in-game advantage. Because of the precedent set up by the other, similar side-quests, the effect is either that the player is annoyed by Kate or else avoids Kate completely.

In the story, on the other hand, Kate is the player character’s main love interest. What’s meant to be an emotional moment towards the end of the game instead comes off as ineffectual because an emotional attachment with Kate isn’t established during the course of actual gameplay. In fact, the gameplay is in conflict with the story, and acts against the story’s attempts at establishing this emotional attachment. If the game had gotten the player to like Kate through gameplay, the emotional moment would have been much more effective. This conflict between story and gameplay in Grand Theft Auto IV suggests two things. Firstly, gameplay becomes meaningful to the story whether the designer intends that meaning or not. And secondly, coherence is an important factor in making gameplay that harmoniously reinforces and enriches the story being told.

Coherence can be as simple as making sure that the units and rules in a World War II themed game somewhat accurately reflect our expectations of World War II. Most game designers are already skilled at implementing this kind of coherence (realism) into their games. But beyond this, designers interested in telling stories better need to start considering what kind of plot they want their game mechanics to create, and whether that plot is telling the story the way they want it to be told.

One example of this second kind of coherence, ludonarrative coherence (described variously by Clint Hocking and Jonathan Blow), can be found in BioShock (2K Games). The relationship between two non-player character types that populate the levels of the game, the Little Sisters and the Big Daddies, is an important part of the game’s story. This relationship is made evident through some of BioShock’s game mechanics, mechanics the player can’t avoid if they want to become strong enough to progress in the game:

1. A Little Sister cannot enter or exit a level without a Big Daddy.

2. A Big Daddy will follow his Little Sister.

3. A Big Daddy will threaten and push anyone that scares his Little Sister.

4. A Big Daddy will attempt to kill anyone that tries to harm his Little Sister.

5. The player cannot interact with a Little Sister until her Big Daddy is dead.

These simple interactions between units, their rules, the environment and the player are coherent with the game’s story. They are a visceral way to showcase the Big Daddies as protectors, demonstrating their unyielding dedication to the safety and wellbeing of their Little Sister. Without saying it orally, textually or visually, these rules procedurally communicate a strong relationship between these two units, reinforcing the story that the game is trying to tell. These rules can be usefully described as dynamic plot devices; together they create dynamic situations for the player to experience the story of BioShock.

No playthrough of BioShock is ever the same, but everyone who plays BioShock experiences the same relationship between Big Daddies and Little Sisters. Without these rules there would be far fewer opportunities for the player to observe and emotionally react to their relationship outside of an initial opening cinematic. Sure, these NPC behaviours are also sound design decisions—decisions which were likely made for reasons that have nothing to do with the story, but that’s the beauty of it: you shouldn’t have to sacrifice good game design for story.

Putting it All Together

So far we’ve discussed the foundation needed to establish a common framework, and the implications this foundation has on the way we talk about and understand stories, characters, agency and game mechanics, but we haven’t really established what that common framework is. The crux of it all comes in shifting our perspective. By seeing all of the various tools we use to tell a story as systems in their own right, systems of meaning that can be used to affect the way a story is told during gameplay, we can refine our perceptions to what matters most about each tool and how they interact when building an experience.

When building a house you might use hammer, it might be made out of steel or zinc alloy, it might be 12” long, it might also be useful for hanging picture frames, and all of these characteristics might be important to the person wielding the hammer, but when talking to the person working the bandsaw, what matters most is how these two tools interact with the raw materials available in order to produce a set of trusses for the roof. Whether we want to impart a certain atmosphere through lighting, a certain emotion through music, a certain mood through character art, or a certain reaction through game mechanics, all of these tools can be used well together if we holistically frame the game and all the tools we use to create that game as a system that communicates meaning through play.

Our task as designers and artists is to use that common framework to visualize a holistic blueprint, to know how to intentionally and purposefully use all of the tools in our toolkit to create a game that tells a story the way we want it to be told, and to understand how one tool interacts with another tool in order to create a game where the total sum of those meanings constructed and expressed by each individual tool comes together to form a singular, cohesive (or ironic) story as experienced through play. With a common framework in mind, artificial intelligence isn’t designed to accurately simulate cognition, but to create compelling story experiences; levels aren’t designed with the back-story in mind, but with an aim to meaningfully reinforce the story being told; and game mechanics aren’t just fun, they’re meaningful.

But after all that pomp, I do want to emphasize that we are already doing this today, to some degree. I hope many of you reading this article weren’t surprised by what was said and saw instead a reflection of your own thoughts. Many of the notions presented here come naturally, others need refinement, and so much has yet to be discovered. We still have a long way to go in perfecting our craft, and I hope the idea of a common framework based on coherently communicating a shared meaning will help get us there.