The Toolbox of Affect

So with Steve Gaynor’s wager, GDC, and the discussion after each, I’ve been thinking about the tools we have at our disposal as designers to create emotional response. Not just what tools are in the toolbox, so to speak, but the biases designers have for and against using some of those tools. Some biases seem natural, and others baffle me. It seems difficult to discuss what sort of techniques to use when, if we can’t even agree on what should be in the toolbox to begin with.

When I wrote in response to Gaynor’s bet, about realism in human faces & behavior being key to the mass appeal of games that convey emotionally complex themes (more mass than today anyway), I hadn’t thought that much about the relationship between those two factors in that way until then. But in writing about it, I convinced myself, you might say.

Interestingly, there were a number of points and questions brought up about this specific topic in Ken Perlin’s roundtable at GDC on Interactive Actors that Express Emotion. Sadly, I couldn’t stick around to talk more and get people’s names, so I can’t credit the people with the corresponding questions, but I suck so oh well.

Is it possible? 

The question was raised if it was even possible for us to create a believable virtual human being. While it’s an interesting philosophical question about how much can we truly understand about our own nature, it constrasts with the unfailing progress our medium has made in the past 30+ years. Having gotten to the point we are at now, there is no compelling evidence that we couldn’t continue in this direction. With one key caveat, to follow.

Do we want to? 

Do we even want to create a realistic human being that you can interact with? When I read or hear Ray Kurzweil talk, he predicts that we will be able to create complete virtual beings, for all intents and purposes function as people. Their intelligence is computationally comparable, that visually and behaviorally they are comparable. To be honest, that kinda creeps me the fuck out. At least the notion that people might turn to virtual characters in full on replacement of human relationships – that is not a future I particularly want to encourage.  (The notion that this will somehow happen simply because the processing power is available also seems naively optimistic, to understate the point).

In other words, creating virtual humans is for suckers. Creating virtual actors, however, is the proverbial hot shit. An actor has to maintain a constrained fiction for a short amount of time, and one of an actor’s main goals is communication (of motivation and intent). This paradigm shift (which has more AI implications that I won’t go into here) is what specifically makes believability achievable, entertaining, and affecting. And ideally avoids the future where we all have virtual girlfriends/boyfriends. (Ok, realistically we’re probably headed there eventually anyway, but diverting the on rushing river is perhaps more feasible than stopping it).

Do we need to? 

It’s not like emotionally affecting media is a new thing, obviously the novel has existed for a long time. Although you could make the argument that reading is certainly less mainstream film, the argument is mainly a financial one (which I prefer not to have as the sole supporting leg in an argument on emotional affect). But that’s beside the point in any case – nothing in games limits you to using or not using that in this regard. Planescape: Torment is easily one of my favorite games. The amount of text in that game is, well, a lot.

That’s what I love about games, we have access to all the tools of all the other media. You just have to be careful to not overuse any. If the only way you have to convey a key plot point is a cutscene, you can do that. The guiding principle here is that you can’t rely to heavily on techniques from other media for fear players get frustrated, but as Planescape: Torment shows, it’s definitely a guideline and not a rule.

The tool of human performance to convey an emotional response is an incredibly powerful one. When that tool is completely in the toolbox, where it is easy to create those performances, then, something that is the opposite of shit hits the fan. That is still years off, of course. 

The strengths (or not) of the medium

Some say focusing on this realism doesn’t play to the strengths of the medium. Now, before I start on this point, let me say, Jason Mitchell’s talk on the visual style of Team Fortress 2 was probably one of the most inspiring sessions at GDC (now downloadable), with its very multi-disciplinary approach. He went from analyzing the artistic styles of American illustrators to the shader lighting equations used to recreate aspects of those styles.

Wandering into San Francisco’s Cartoon Art Museum the next day, I was equally inspired. The art being shown there there, from the work of Mary Blair to feminist political cartoons, just sent my mind creatively reeling in all the possible directions you could pursue for non-photorealistic visual styles. That is certainly one of the most open areas for exploration in games right now.

But to say realism doesn’t play to the strengths of the medium is incorrect. It doesn’t play well to the strengths of the technology, today. The strengths and weaknesses of the medium apply no stylistic constraints. Interactivity & simulation have no inherent art direction. As beautiful as the visual style of Team Fortress 2 may be, I can’t see those characters conveying more complex emotions such as regret or passion. Norman Rockwell’s characters (referenced in Mitchell’s talk) are certainly capable of conveying a specific set of emotions, but not others. TF2’s character are the perfect tool for the purposes of that specific game, and other games need other tools.

I don’t understand why there’s a natural distaste for this particular tool compared to any other – the more the better, in my book. And when it comes down to making the game you want, any tools you have, or even dirty tricks, are fair game in my book, if they create the desired response. When we have so rarely achieved the range in emotional responses that games are capable of, can we really afford to be so picky?

Diminishing returns 

Do we run the risk of hitting diminishing returns, as Jesper Juul asks? Certainly a lot of work goes into effects like ambient occlusion crease shading, and even more into all the animation systems needed to create believable motion. Five years ago, the same amount of effort certainly brought more drastic results. 

Does that bely lack of forward progress? If we’re traversing the slope of the uncanny valley, the distance we travel down the slope is longer than our horizontal movement. That doesn’t mean the work isn’t moving towards achieving the same goal, it just means it is difficult to pinpoint the impact of these effects while looking at them individually.

On most games I’ve worked on, we were building our own engine at the same time, so rendering features would go in slowly. Each individual one might be alright looking, noticeable but not wowing. Inevitably, as development goes on, someone takes a current screenshot and sends it out with an old one. Looking back at early playable demos with relatively untextured, poorly lit, unshadowed levels to a near-finished look always drives the point home. It can be easy to underestimate the holistic impact caused by such changes, without that perspective.

Holism, formalism and sensual…ism?

The lack of holism is problematic for me in this sort of discussion. I enjoyed Clint Hocking’s talk at GDC on immersive fidelity (also downloadable). In part he discussed the differences between formal immersion (stemming from game systems) and sensual immersion (from audiovisual stimulation). However he left it as an open question at the end as to how we can best combine techniques in these two areas to greater effect.

While it’s a very useful point to make, I can’t help but wonder if the underlying assumption is somewhat flawed. By looking at these aspects individually, they already begin separated. Granted, you have to break them down like that just to begin to understand the complex factors involved in each type of immersion. Once you do, can you continue to treat them separately and still successfully combine them? Or do you have to synthesize that understanding into a new perspective that treats boths aspects holistically?

A while back (an exceptionally long while back in internet time), Tadhg Kelly pointed out what is also a pet peeve for me: we all make fun of game designers that really want to be making movies, but those that really want to be making board games are just as confused. So treating techniques as belonging in the formal category vs. sensual may not end up being helpful. That breakdown may help simplify how you think about making one mechanic or performance more immersive, but relying on it exclusively seems to unnecessarily constrain how you think about the combination. Unfortunately dealing with the combination is a much more intuitive affair I can’t necessarily yet convey, I’ll just have to be critical here with contributing a more useful insight. Again, Isuckohwell. 

One of the points the roundtable came to generally agreeing upon (and I didn’t want to follow up on for fear of belaboring a point) was that creating a meaningful relationship with a character should primarily be driven through mechanical impact – how your relationships with characters change how characters allow, disallow or alter game mechanics. Which is just a bit weird. Now, to be clear, it is the most underused and overlooked aspect of creating emotional relationships to game characters today, and as such, is definitely the area to explore for biggest immediate effect (and affect).

Yet to dismiss the sensual components of our physiological responses to characters is like throwing the baby out with the bathwater – before you’ve even washed the baby. Completely ignoring those components is one of those painful lessons we might all be better off if we didn’t have to go through, the kind that often results in fantastic mistakes much the way Hocking points out Trespasser. That game using a IK driven player animation system to attempt to create immersion with one-to-one correspondence of player input to avatar actions – but by failing to simplify how you interact with the physics simulation at the same time, your interactions end up being much more complex than they should have been (and not immersive).

If you say either formal immersion or sensual immersion has more of an impact than the other in general principle, is that attitude really going to bring about meaningful change to our medium? In any individual game the balance may be shifted by design, naturally, but I don’t think actively prioritizing one over the over can lead to a useful basis for constructive analysis. The real strengths of the medium come when we are no longer watching movies while playing board games.

Taking these points to a silly extreme

These biases aren’t limited to designers, though. I’ve been slogging my way through The Transmission of Affect, by Teresa Brennan. She rails against the psychological bias on transmission of affect through visual means, and also its cause – our society’s reliance on the visual sense. She argues (so far anyway, I’m about a third of the way in) that the predominant factors in the transmission of emotions between people involve other biological processes, hormones, pheromones, and so on.

Which is unfortunate for two reasons. One, because I bought the book hoping at least some of it would analyze the pyschological processes of transmission of affect through media – especially given the mediated nature of the image on the cover, it’s misleading to say the least. Second, it’s ignorant at the same time. While those physiological processes may be primarily responsible when face to face, it’s obvious media can in fact encourage emotional response through purely audiovisual means. Although it does open the question of what other tools will we have when we have access to smell-o-vision…But god help me if I ever have to debug that.

Once more, with feeling

In all this discussion of creating emotional impact, we do run into the issue Leigh Alexander raises, like actors do in creating their performances, especially early on in studying acting, that of over-compensating. This point, which is raised in various guises on this theme, I’m always a bit conflicted on. The same argument is often taken to an extreme against games that have serious themes – because they have typically exhibited such ineffectual, over-wrought attempts at delivering their messages, it must be that the expending of effort itself is flawed.  While she doesn’t make this point, some often extend the argument to say we shouldn’t even make the attempt.

The clumsiness is not related to effort of course, but overly applying conscious or analytical thought processes, part of the natural progression of learning any skill set. Like acting students perhaps more effort is required, but in different areas, to advance enough to convey what one really intends. When you start out you have to apply conscious thought processes, and your performance is impacted negatively. As the skills become automatic, you perform the same processes instinctively, freeing yourself from the burden of over-analyzing yourself as you learn.

It seemed like the Portal  team tried quite hard for their game to have an emotional effect on players (or, more accurately, you might say they explored very hard). Not from playing the game, but from the stories of what they did during the game’s development. Most products of design or craftsmanship appear the most effortless when they have been slaved over incessantly.

The root of the argument, though, is key – keeping the sincerity of feeling in your attempts. Occasionally you read op-eds that call for designers to make games not just for themselves, in order to broaden the market. This sort of thinking is confused – the more useful advice is that whatever project you find yourself on, you have to find something in it that truly appeals to you, even if the game on the surface wouldn’t strike you as interesting. At the simplest level, you need to feel what you want the player to feel – if you don’t, you’re in trouble.

As usual, I wrote way too fucking much. Hope you made it to the end.