Digital FX expert Joe Letteri explains WETA's groundbreaking work on James Cameron's 'Avatar'
Last Friday, I got up incredibly early to go watch a bunch of four-year-olds in bathrobes sing some Christmas songs and carry around a rubber doll they called Jesus. And as low-tech as it was, it was absolutely thrilling to each and every parent in the room. Toshi may not have remembered all the words to "Away In A Manger," but he bellowed every one he did remember with all the heart any parent could ask. As soon as that was done, I was out the door and in my car and on my way to Burbank, where I sat down for coffee with Joe Letteri, a guy I've been waiting to meet for a while now, so we could talk about the absolute opposite of that Christmas pageant, the mega-expensive and cutting-edge technological marvel of "Avatar".
As FX supervisor for WETA Digital on "Avatar," he's been buried in secrecy for the last five or six years, and that whole time, I've been itching to sit down and ask him about the work he's been doing. Since Letteri was also a key player on "Lord Of The Rings," one might argue that there are few people in the business better suited to talk about world building on a certain scale right now. As soon as we had our drinks, we sat down outside and I turned on my tape recorder:
DREW MCWEENY: I grew up watching these films. Like I said, I was seven when “Star Wars” came out.
JOE LETTERI: Yeah, yeah.
DM: It rewired me. I staggered out of the theatre and said to my parents, “All right. Who made that?” We looked at the poster and they were like, “Okay, written and directed by George Lucas”. That was the first movie where they really did the big behind-the-scenes thing, where for a year we saw specials and magazines and it kind of lit the fire for a lot of kids my age. It basically said to us, "This is a craft. You can learn to do this. It’s not something that’s magic or impossible."
JL: Well, it looked like people climbed into space ships and took off. That was the thing.
DM: Right. It’s like, "Whoa, that’s what I want to do," you know? At first, it was "I want to climb into a space ship and fly somewhere," but it quickly became, "I want to make movies that take you to other planets that don't exist." And George made it look like that. This is the first time in awhile where I have… you stop thinking about the process at all when you’re watching, and I know that was a goal. I know that that was something that as far back as… I think I read the scriptment in ’97..?
JL: Yeah, that early treatment, yeah, yeah.
DM: At that point it was like, "He’s nuts. There’s no way." I don’t even know how you begin. At what point did you seriously begin the process of, "Okay, this is really happening. We’re really going to have to pull it off now and how do we marshall the forces?"
JL: So okay, what happened was we finished “Lord of the Rings,” right? We came up here and we had this big night at the Oscars where we got that sweep that night for “Return of the King”. The next day, Peter calls and says, "Jim called. He’s got his new 3D camera system. You’ve got to go take a look at it." I said sure. So we started talking to Jim about 3D movies because Peter wanted to shoot “Kong” in 3D, and we talked to Jim about using his camera. "Where is it? You know... is it production ready?" He talked about the documentaries he’d shot, but obviously for a big film like "Kong," we'd have to have multiple package ready. He said it was probably not going to be ready in time, so Peter just said, "Okay, we’ll shoot 'Kong' in 2D," but he really wanted to do it in 3D. Then we just stayed in touch, and as we were working on “Kong,” we talked about what we were doing, and how we were going to have a lead character with no dialogue who’s still got to be able to carry scenes and play opposite Naomi, and we were still trying to figure all that stuff out. We told Jim we were planning to do a facial motion capture, which was new because Gollum we did without any motion capture for the face.
DM: I think a lot of people think that Gollum was all motion capture, but a lot of it was key frame animation, right?
JL: Most of it was key frame.
JL: Andy was motion captured for sure, but a lot of it, he was done on-set, and so it was matched to what he did on-set so he could be put in place. For the stuff where you see him photographed with everybody, there’s no motion capture.
JL: You couldn’t do on-set motion capture at the time. We did it, I think, in two scenes in “Return of the King,” but for "The Two Towers," there was none. It was all key frame. We did have a motion capture set and we did work with Andy to mo-cap some of the scenes, so I shouldn't say it was all key frame. Some of it was motion capture, because we were able to look at the raw data you mo-cap and use that as animation reference, but you can put him into it in a few moments, so there was... maybe I’d say 25% was motion capture. That went up a little bit higher on “Kong,” but not much. Maybe, you know, 50% or so. And we upped it more on this one, but it’s certainly still not 100%. That whole idea that motion capture is an automatic process... it’s an automated process for sure, but it’s not automatic. It takes artists still looking at every frame of this stuff and saying, “Is it right? Is it right?” You know, we build systems that are smart so you can say, "Okay, that shape on the mouth in that particular bit of dialogue is wrong. Let’s fix it." Then you can put it into the system so the next time we see that shape on the actor's face, you know what that's supposed to be and we get it right.
DM: Oh, okay.
JL: It still requires artist's intervention because as much as we try to understand what’s going on with the human face, there’s still just a lot of unknowns. Even the way the muscles are laid out. They’re not even known to… you know, we’re working with a lot of biologists and bio-engineers on this stuff. We’re really just starting to crack how that is. And even so three years ago when we started, it was less known then. So what we do is we take it from the outside in. There’s a system we use called FACS. I don’t know if you’re familiar with that at all.
DM: I’ve heard of it.
JL: Facial Action Coding System. It was developed by a guy, Paul Ekman, at UCFF years ago to try to figure out if people are lying. What causes people to make a split second decision? Like why would a policeman pull a gun on a suspect and decide to shoot him or not shoot him? What kind of clues tell these guys, who are really good, what it is when they know somebody’s bluffing or not? And what he did is he broke down the face into what he called action units. And so he broke down every part of the face and you can see what the response is for any particular emotion.... any set of muscles that work together... and that’s what we did for Gollum. We broke it down that way and we actually used the same kind of tricks. It was really subtle one-frame self-expressions, like when he’s lying, that you can just see if you frame-by-frame it. You can see that that’s what’s going on. That’s what happens in the real world but it just goes by so fast that you don’t really ever notice it.
DM: The moment where Gollum came to life, as I was watching "Two Towers," was the moment where he has his schizophrenic conversation. That’s where I went, "Alright, he’s alive. That’s real. I buy it. I’m totally into the character."
JL: Yeah, yeah.
DM: And I think with “Kong”, the jump there was that even though he was totally non-verbal, we still understood everything Kong was thinking and doing.
DM: Which is beautiful and it really, to me, was a performance.
JL: Oh, yeah.
DM: The work between the animators and Andy and the way it came together, it felt effortless... like Kong lived. I really feel like you guys have taken it to the next level, though, this time. Different actors have… I’ve spoken with two guys who really advocated for the process after they went through it. Both did it for Zemenkis, but they… Gary Oldman and John Malkovich, guys who you think would be really opposed to this kind of technology and things like that... but it's the exact opposite. They were both like, "In a heartbeat. I loved it. I loved the freedom of it and I loved the feeling that I can play anything." For Oldman, the idea that he played Tiny Tim and Bob Cratchit blew his mind. And he loved it. So I think that actors, as they embrace it, it seems like a different skill-set. Like you kind of have to relearn some of the tools to really be effective in performance capture?
JL: You kind of do, but it’s interesting talking to some of the actors I’ve spoken with about it. You do try to get everyone’s take on, "How’s it working for you? Is there anything we else we kind of need to do?" And they’re saying it actually works for us because it’s like, to them, it’s like the old days of coming up and learning the craft. You didn’t have a set, you just had an empty room and a bunch of people just acting and that was it, you know? Before you got famous or got a job, you didn’t have costumes. You didn’t have anything. You were just acting. So to them acting in a room just strips it down. And it’s actually great watching these performances because it’s like there’s nobody in the room and you’re just watching these dramas, you know? It’s really cool.
DM: I have to say the performance in this movie that I think is the jump, the one where you look at it, and you’re like… she just looks alive...
DM: Right. What Zoe does in this movie is unbelievable.
JL: Yeah, yeah.
DM: Everybody’s very good. Joel David Moore seems to really take to it. Sigourney’s amazing. And I love how much it’s Sigourney to the point where the audience… the first time she walks out, they laugh because they’re like, “Holy crap. It’s Sigourney Weaver... and she's blue!"
JL: Yeah, yeah.
DM: There’s something about the way Zoe does it and about the way she slips into Naytiri’s skin that is other worldly. You really buy that she’s not human and that the physiology works differently, and she makes it really beautiful. As you guys are watching these performances come together, do you have moments where you go, "Wow, I didn’t think it was going to look like that"?
JL: There’s a few of them, but you know really what happens is... like, you’re approaching this stuff incrementally to the point where… the problem with CG is until everything is right, it just doesn’t look right at all. So it’s almost like doing a puzzle where you’ve got this thing and you’re putting it all in and you’re layering it and you get it piece by piece and it’s in there and the final piece is there and you look at it and say it’s done... great. Okay, onto the next one, you know? So for me, it wasn’t until the premiere that I got to sit back and actually just watch it, which is great.
DM: Peter was talking about one of the things that he’s very proud of with WETA, and one of the things he says is proprietary and that he feels like you do better than anybody in the business... it’s the eyes. To me, that is the key to your characters. So much of it comes down to if the eyes work, and the design of the eyes seems to be cheated a bit so that you have that much more surface space to play.
JL: It's cheated hugely. It’s actually a very similar idea to what we had with Gollum, which is one of the things we talked about with Jim, you know, getting back to sort of the early days. These eyes are almost 3 times as big proportionate as human eyes. You just don’t realize that when you’re looking at them. We did the same thing with Gollum. His eyes were like 2-1/2 times human size, and you just don’t realize it because you’re so used to wanting to see a face that you really have to exaggerate it to make it work. But the Na'vi also had the sort of cat-eye design so there was a little bit of that while still trying to make them look human, so again, we worked on those eyes for a long time, from the initial design to first animating them and showing Jim, "Okay, here’s what your design was, but this isn’t going to exactly work because we need to get his expression," and, you know, it was just an iterative process for us to figure it out and then to talk through with him. "Okay, where do you want to go with it?" Because you’re… obviously, the easiest thing to do is take it all the way back to human but that’s not what you want to do, so you’re always just trying to find what’s the balance between getting this character that you want but still getting all the human expression that you need.
DM: What is it about the WETA eyes? Obviously it’s not just a software thing, but it’s something in how you design or something in how you execute them that I think takes it that extra distance from an effect to something that reacts the way we believe an eye should. How much of that was a process of you guys having to go through sort of trial and error before you really looked at something and went, "Okay, that works. That sells it."
JL: It wasn’t exactly trial and error, but it was a process where we knew what we were going towards and it was getting all the pieces to work. You break it down. For example, there’s the… you’re trying to capture the performance of the eyes. We have an optical system that will track the eyes separately from the rest of the face to figure out what they’re doing. Then the animators have to look at that and just say, “That’s great. Does that really come through or not?” And if not, you’re doing animation on top of it to get the detailed movement of the eyes, the glances working the right way, things that are off just because... okay, what Sam did was this but because Jake’s character’s eyes are this far apart it makes him look a little bit like he’s looking in the wrong direction… whatever. There’s lots of detail that you go into for every shot just to get the animation going. You’ve got all the movement of the skin around the eyes and the muscles that contribute to it. When the eyes move, they have… they influence everything else around it and they’re influenced by everything else around it. So you’ve got all that as far as the creature and muscle side of it goes. And then you’ve got all the lighting side. The skin, you know, getting the thinness of the skin to work right, the subsurface... especially with the blue skin. That was tricky, to not really make it look thick because we needed a thin layer of blue and we still needed a blood layer underneath it, so for the whole skin getting that translucency was much trickier because of the color. And then the eyes themselves because they have a certain translucent quality to them. They also have this kind of caustic quality to them because they’re lenses and they focus light and that’s a lot of what you see on the eyes... how they focus light. Well, we wanted to do this whole thing on this movie, to really work with global illumination because we wanted everything to be lit in the same world. The characters, the jungle, everything. It’s like just lighting… look around you. It’s like lighting this whole patio and the parking lot beyond it, and the street out front... and everything else. So we had a whole project going to get that working, but of course that all has to come together in the eyes. You have to figure out how to gather all that light and focus it into the eyes. So we knew we needed to do all that but it just took awhile, but once it came together you could see it was working. And it felt like, "Yeah, okay. We have it."
DM: When you’re building a world completely as you did with Pandora, where you’re starting from zero and everything, the ecosystem and all the life forms and all the plants and everything has to be built... is it more difficult to do that than what Peter did with “Lord of the Rings” where you’re mixing effects elements and you’re combining locations and things that are built? Because it almost seems like if it was a completely CG environment you’d have more control and you'd have more freedom to do things.
JL: Which is a bad thing.
DM: Oh, okay.
JL: It is. Because when you have something, like when you’re putting a character like Gollum next to Elijah, you’ve got your reference right there. He’s got to look as good as Elijah, performance wise, skin wise, everything. He’s got to look totally believable. So you’ve got your reference right there to work with. And it just hangs together. When you don’t have that, you can just start from scratch and put him out there, and you can miss the mark and not know you’ve missed it. But eventually you’re going to see that because it’s just not going to feel real to you, you know? So you have to keep all this in your head about just trying to figure out what are the pieces that are real and what are the pieces that are not real. And it would have been really hard to do this movie without the experience of having gone through "Rings," you know, and then working with Gollum in the plates and then Kong, where it's the same kind of thing but now we’re building more of the environment around him and then taking the step to just, okay, now we’re building the whole thing. Because you keep in mind all those things that you learned from doing all those other films on what actually you’re looking for to make it look real and just don’t forget it because you don’t have the reference anymore.
DM: Interesting. The first test they showed me at Lightstorm of the 3D was a couple of years ago, and they showed me some of the stuff they’d done where they retrofitted other films, films that had been shot in 2D. And the one that really blew my mind, the one where I started to get really excited about the potential of “Avatar,” was “The Two Towers”. It was a test with Sam and Frodo and Gollum. And what did it was seeing Sam in the foreground, Frodo in the background, and Gollum between them. That’s where my brain just went, "Alright, well he’s holding a space. He’s real, because there he is standing between two real people."
JL: Yeah, yeah.
DM: So much of this does take place either in the Na'vi world where it’s all Na'vi or it takes place inside the human buildings, but there are those moments where things cross over. I think particularly striking is the moment where Neytiri finally meets Jake.
JL: The human Jake, yeah, yeah.
DM: It's beautiful, and by that point in the movie, you want to see them together and then you realize how the scale is so different, and it’s a mind boggling moment. It really sells Neytiri. I think that’s what does it. You go, "Oh, well, there she is and there’s Jake and they’re together and wow."
JL: Yeah, yeah. That was a whole design thing because we talked about this with Jim a lot. Early on you’ve got, like I said, you’re in the human world. You see the Avatar. You introduce the size of them in the tank. You see how long they are. Then you see them in the ambient room where they’re waking up. You get to see them next to human reference. Then they run out and they run past a few med-techs, you know? After that, you’re pretty much out in the jungle with them, and at that point we knew that you would just be looking at them as human. You forget about their size because you’re doing the close-up, you’re doing the 2-shot... whatever. You’re just going to think about them as human. It’s the exact same thing we went through with the Hobbits. Once you’re in a scene with just the Hobbits, you never thought about their size because you had no scale reference. So we knew the movie was going to take that arc and it needed to be established early on, and you needed to bring it home at the end. Not only for scale reference but emotionally. You really need it to see who he really was.
DM: That seems to be the big jump is combining these photo-real characters and humans to an extent that you stop thinking about what’s real and what’s not. There are moments in “Avatar” where there are human characters who, if I’m not mistaken, have been digitally duplicated.
JL: Oh yeah, there were a lot of digital doubles in that movie.
DM: That’s fascinating. I think that’s one of the things that is one of the most unusual effects in the movie because you’re looking for the aliens. You’re looking for the big things but those digital doubles... that’s a whole different game you guys are having to play. Can you talk about why you used those, when you used those, and sort of how they were developed?
JL: Generally we use digital doubles for stunt-work. Things that are too dangerous to do otherwise. Sometimes you use them for things that are too difficult to shoot like those shots of Quaritch fighting Jake at the end in the EVA suit. Probably half a dozen of those were digital double shots, you know, because you just couldn’t get the right camera angle on them to shoot them. It all comes down to the same kind of technology, the same idea that we started with with Gollum really. It’s just what’s happened over the years as Gollum took a year to make and all that time and effort going into one character. Now you have to do like dozens of characters, so you really have to take what you learned and apply it, and we really learned a lot about skin in an interesting way by doing blue skin because like I said it was really hard. Blue skin just wants to look like plastic. It really does because you’re absorbing all the light and you’re not getting any of the red back that tells you it’s flesh. So we had to come up with another technique to figure out how to let the light go back in and come back out as red because we knew we wanted them to feel fleshy around the eyes and the lips and have red blood and around the hands and everything. But it really focused our effort, too, on skin to the point where it’s like, "Okay, we’ve got a pretty good idea how to do human skin now." Too bad we didn’t have to do much of it, you know? But we did it for the digital doubles, so we treat everything of a piece. Whatever you’re doing… for us doing a vehicle or skin, it’s all the math behind it. All the science behind it is all based on the same thing.
DM: How do you create a sense of scale in some of this because, for example, the Home Tree sequence... unavoidably if you watched 9/11 happen, there is that same sense of scale of something coming down. It is gargantuan and in the theatre it’s not just the sound, it is the imagery and what we’re seeing that really sells the fact that this thing is enormous. But you are dealing with digital models, so how do you sell that idea of scale? What are the things that you guys… the signifiers that you use?
JL: It’s a real balancing act and it really depends on the object. Like for example, the Home Tree being a good one... if you’re sort of in close and you look at the bark on it, you expect the bark to be sort of like a tree bark, right? And that makes sense, but once you’re far away at a distance, if you really had that scale of tree bark, it would be so small that from that distance it would just look like nothing. You wouldn’t get the tree bark scale. So what we have to do for something like that, where we know we’re going to use it in multiple scales, is work in multiple levels of detail. In that case, it looks like tree bark, so when you’re in close you can see what looks like you’d expect, what looks like tree bark, and when you’re farther away it still looks like tree bark. When you’re even farther away, it still looks like tree bark. There’s just multiple scales of detail. We do it with the dressing, the leaves and the foliage and all that sort of thing. Sometimes we have to play with the size depending on the shot. When Jake and Neytiri are up at the top of the banshee aviary and they walk out and you see them surrounded by all those leaves, you know you have sort of a good size for them to believe that it’s a large tree, but even at that size once you’re down on the ground and those leaves are 1,000 feet in the air, they’re just going to look like little bits of cotton. Sometimes we just have to play with it for different shots just so you understand what it is you’re looking at, so the tree still looks like a tree.
You can read the rest of this conversation in part two of the interview.