Digital FX expert Joe Letteri explains WETA's groundbreaking role in 'Avatar,' 'The Hobbit,' and 'TinTin'
The conversation continues here exactly where it left off in part one of this interview.
DM: Wow. Just throwing one of these technical challenges at you would make this a wildly difficult film, so I really can’t get my head around how you guys broke everything down. One of the thing I’ve always hardest to pull off in effects is flight. I think flight... there’s just something inherently fake about it on film.
DM: I would say Terry Gilliam’s “Brazil” got flight right, and for me Miyazaki in animation is one of the few guys who really gets it.
JL: Sure, yeah.
DM: I think he will love this film. How did you approach that?
JL: There’s a clue there in what you said. Miyazaki has the edge. If you’re not bound to what you’re doing physically… if you don’t have actors on wires, if you really can just make the characters fly and react to gravity the way they should if they were there, then you do have the ability to do it. That’s one of the freedoms that you do want to have by doing it this way. And that’s really what we were able to do because we weren’t locked down to pick points on a harness or anything else that normally defies what you’re trying to do by putting somebody on a riding rig.
DM: It’s incredibly effective. Is it a different process for you working in 3D? Are there different things that you have to accomplish? Or is it the same basic type of challenge and then just a rendering issue?
JL: There’s a couple of things there. If you’re looking at the 2D like the live-action side of it, yeah there’s a lot of harder work that goes into it, because things that used to be simple… not necessarily simple, but you know more straight-forward... like cleaning up wire rigs and things like that... in stereo, those are much more difficult to do because every pixel is a different point in space and you have to account for that in both eyes. Over time, of course. So that gets harder. But from a straight rendering point of view, it gets… there’s more to do but conceptually all you’re doing is rendering everything from two different cameras. Technically it’s actually not that hard other than figuring out how to deal with all that data. There’s sort of a pipeline issue that goes with it, but what does become tricky is compositing because the idea of just having the thing simply layered right now goes away because that could change in both eyes. What we had to do is we came up with a new compositing system where everything is now compositing into 3D. We restore all the depth information for every pixel of every element that we create. And we confine it in 3D. Because it was just the cleanest way to just make sure everything worked in the world correctly. So all those little cheats we were able to do before, you know, like painting in a shadow fix or adding a bit of the smoke cloud or something? Most of that goes out the door, except for a point in the deep background where you’re not worried about pixel issues. You can use things like that. You can use matte paintings and photographic elements, but most of it has to be done by compositing actual three-dimensional elements now.
DM: Now that you’ve built Pandora, now that you guys have created all these assets, would it be easier to do a sequel?
JL: If it was shot in the same location, yes.
DM: Okay. So stay away from the other parts of Pandora…
JL: Yeah, as long as he doesn’t take us to the desert side of Pandora… (laughs)
DM: But for the Na'vi themselves, you guys… would that be something that now that you’ve solved a lot of those challenges or solved a lot of the big questions, jumping back in, would you…
JL: We’d redo it all.
JL: Yeah, because I’m sure whatever it would be if Jim and Fox decided to do it in a year or two years, we will have learned something else about something that we’re just going to want to do differently because we like to just keep everything moving along in parallel. As much as you like to say this... put it this way… there will be stuff we’ll use. A lot of the assets and trees, you’re right, are built. There’s a lot that we wouldn’t have to do but as far as what we’re doing with the muscles, the faces, the eyes? I’m sure we’ll have new stuff by then that we’ll want to put into it.
DM: How much of the decision to make “Avatar” instead of "Battle Angel Alita" came down to what was ready or what was possible at the moment?
JL: I don’t know. That was really all Jim’s decision. He just came to us and said, "I want to make 'Avatar.'"
DM: Oh, okay.
DM: I didn’t know if that was a conversation that he had to have to see if he could even pull "Alita" off, because I know that as a lead she would have to be completely created.
DM: It seems like he has almost every time out... and I kind of love this about him... like he almost always makes the most expensive film ever made. He almost always has people thinking ahead of time, "Oh, it can’t possibly work," and he almost always sort of throws some challenge at himself that seems insurmountable.
DM: For you guys, is that the kick of working on a Cameron film? Knowing that going in you don’t know if you can really pull all of this off?
JL: No, we knew we could pull it off, because we wouldn’t have taken it on otherwise. Jim wasn’t going to settle for less than what he wanted. So, you know, we knew going in that we could do it, but it was really figuring out how. I mean, you read that scriptment. I read that, too. It’s like I want to see this movie and I guess the only way its going to happen is if we make it, so let’s figure it out, you know?
DM: I’m intrigued by guys that insist on pushing things forward, because it seems like that’s a risky place to take your career, film after film after film. And the payoff is something that we really haven’t experienced before. When WETA is choosing what they’re working on, I know a lot of it is driven by Peter, especially as you guys are gearing up on "The Hobbit”...
JL: Oh, absolutely.
DM: ... and that pretty much will become all-encompassing now for several years. Within WETA, you guys obviously have different units that work on different things or specialize in different things. How much of the development of each of these… whether it be skin technology or whatever else... how much is driven by the particular project, and how much is just research that is ongoing? As people come to you, do you have solutions, or do you have ideas about how you’re going to approach them?
JL: It’s a little bit of both. In “Avatar”, you know, again, you’re right. Jim is always in the right place at the right time. “Avatar” allowed us to push forward in a lot of things that we really knew that we wanted to be doing anyway because we just knew to really make these kinds of images, here’s what we have to do. There’s still more…we’ve got to crack the skin. Or we’ve dealt with the eyes. The muscles still aren’t working right. We’ve got to do global illumination. It’s just like all these things that we knew that physically we didn’t quite have solved, we were just… we wanted to move on that anyway. We would have tackled any one of those or any combination of those on some part of the next film. But with “Avatar,” it’s like, okay, now we get to do it all, you know?
DM: Do different filmmakers have a different vocabulary with you as they come in and different levels of how much they understand or how comfortable they are? I mean, I’ve heard about filmmakers who, when they start a process like this, they’re basically learning it all.
DM: And then you have guys like Peter who has been so much a part of WETA’s development or Guillermo who has very specific ideas about how he wants to do things...
JL: Yeah, yeah.
DM: How do you develop a shorthand with a filmmaker or how do you develop the language of the filmmaker that will get them through each project?
JL: What we’ve always done is just worked physically as much as possible. You know, everything that we do is based on something in the real world. If you’re talking about where you want to move your light or how bright you want it to be, you’re talking in F-stops, you know? It’s just that everything is translated from how you shoot on-set, so it makes it much easier to just get through the process. If a director wants to get into specifics of how we’re solving certain muscle issues or flesh tone issues, that’s great. Some will and some won’t. Jim’s definitely interested. Like when we told him, "Okay, we’re getting away from the old-style way of doing the traditional CG muscles of these sort of balloons under skin. We’re going to do final element analysis. We’re going to actually solve the bio-mechanics for the muscles." Jim wanted to know about that because he’s into that kind of stuff, you know?
JL: But that’s not necessary for him to be able to direct a character, you know? It’s just something that with Jim we could talk more specifically about what’s working and what’s not working.
DM: Is it easier when you’re dealing with a guy like Cameron since he came out of an effects background and a design background and is so specific?
DM: Does that make it easier for you? Or do you ever run into a situation where somebody has such a specific thing that they want and maybe that’s not technically feasible? Like how much of the design is form and how much of it is aesthetic? Or how much of it is function?
JL: Yeah, that’s a good question. It’s always still a balance because the way this project was set up you had this template idea. Jim was working on the stage. He had low-res models of everything that he could move around and try to design and get everything worked out. We took the big markers in that as graphically and compositionally what he wanted. And then once we turned that into the high-res models we saw if it still worked or if it didn’t work. If it didn’t work, we tried to kind of bring it back into line with what he wanted. If for some reason that still didn’t work, we’d just go to him and say, "This isn’t going to fly."
JL: (laughs) "It’s just not coming out the way you thought it was going to." And he’ll look at that and say, "Okay, let’s try something else." Or we’ll suggest something else.
DM: One of the big questions that I had about the 3D before I saw the film was how Jim’s cutting style would work. It's so aggressive and has always been so specific, and I think he’s one of the best action choreographers and editors in the world. But it doesn't immediately lend itself to 3D because your eyes….
JL: That’s right.
DM: … are focusing on different things and on the cuts, you’re going to… how did you guys solve that issue of not having the audience laying in the aisles after that first big Thanator chase? With focal length or what? Because that had to be one of the biggest technical questions going in.
JL: Yeah. Obviously Jim had a lot of experience with that coming into it because of all the documentaries and things he shot. One of the things he was doing with all these documentaries was just learning the techniques of 3D filmmaking, obviously. And we had long conversations about it, so we felt we understood what he learned and what he wanted. But he would do a first pass on that anyway. When he turned over this template over to us, it had the stereo cameras already worked out. So they would come in and we’d have everything there. Once we started going through the animation and up-rezzing everything, if we saw something that we felt like didn’t work because of the motion blur or the speed of the pan… we’ll flag that stuff to him and again, we’ll make suggestions. We’ll try a different lens. We’ll show him some things with a different camera move or put the focus in a different place. And of course, as the cut changes, that’s going to change how you use stereo as well because like you said, you’ve got to lead the eye from shot to shot. So Jim was very aware of that and we just learned it. And we just kind of… we just tried to like always look at that ahead of him so that when we were presenting stuff to him, we could flag the problem so we weren’t waiting several iterations to get feedback on it, you know?
DM: Right. Yeah. It really seems like that is the most invisible impressive thing about the film. I never felt worn out by any of the visual sequences. And my eyes... because I have come out of 3D movies and had that feeling like, "Oh my God, was I just poked?"
JL: Yeah, yeah.
DM: There’s none of that. It felt so clean and it really felt like he had a handle on how to keep your eye in the right place.
JL: Yeah, yeah.
DM: It seems like such a different discipline above and beyond acting and telling a story and everything else, and just to juggle all that... I’ve always heard Lucas say that he was going to be an experimental filmmaker after he finished a certain number of films. And he’s talked about it for so many years, but he hasn’t done it yet. It seems like Jim really did just go become an experimental filmmaker for awhile.
JL: He did, yeah.
DM: Are there things now that he’s developed on this film that you guys are going to take with you into other projects that you’re doing? Or that you, on this picture, picked up or innovated or broke ground on?
JL: It’s a little bit of both because just the whole idea of doing this virtual staging, this virtual capture. That was just brilliant because all the pieces were out there. You know, we’d done motion capture before. We’d done real-time motion capture. We’d done facial capture. We’ve had a virtual camera out there, but not on the set that we would just think about... "Okay, Peter go direct the shot this way," because we were always working to plates so you didn’t need it, you know? But Jim’s idea of like, "Okay, I’m going to put myself in the middle of this virtual world," you know? Rob Legato came up with this virtual camera that they designed. It’s brilliant. It’s a fantastic way to work because any director that needs to work in a 3D realm, and I think a lot will be these days, it’s not that big a leap for them to pick up this camera and just become comfortable with it. "Yeah, there are my actors. I see my characters." It’s not a huge conceptual leap. So I think you’re going to see more and more of that happening. No doubt about it. I mean, we’re already doing it on “TinTin”. We shot “TinTin” using the same kind of approach, and it was really easy and really fun to do.
DM: I’m really excited about that because I think people… I think people have a misunderstanding when they hear the term performance capture that it will all look like a Zemekis film, or that it will all look the same as something they’ve seen before. But those are style choices. That has nothing to do with performance capture. “TinTin” to me seems like the one where we’ll really break some ground and see an outrageous style, that Herge-designed world, but with people performing it. I’m so glad that they never made it live-action. I know Spielberg had talked about that so many times over the years.
JL: Yeah, well I think there was exactly that problem… not to get off the subject here… but I think Peter went to him and said “It’s going to be hard to do live-action. Why don’t we just think about making the characters look like the characters? We can do that now.” And Steven saw some of this stuff and went, “Yeah, okay.”
DM: So it really frees up not only the actors but the filmmakers to push us into things we’ve never really seen visually or places we’ve never really been.
JL: Well, I think Jim proved that big time on this one, so yeah.
DM: Well, it’s extraordinary work, Joe. I really appreciate your taking the time this morning.
JL: Oh, absolutely.
DM: And I wish you guys all the best with it, man. I really hope it is what I think it’s going to be.
DM: I can’t imagine it not.
JL: Yeah. Like you said, it’s such a cinematic experience. That’s been the really cool thing... and now people are talking about Best Picture and the dramatic performances, and I think with all the hype going into the technology and everything beforehand, I think people miss that, but that’s what we were focused on all along. We were focused on Neytiri and Jake, because we knew that was the movie and everything else is really dressing, you know? And that’s where we put our effort and then just built the world around them.
DM: That’s what I love about the work you guys do. When I think of WETA and why I think of you guys as the best effect house in the world, the reason is because the characters that have come out of the work you’ve done are so indelible and they feel real to me. Gollum and Kong and now the Na'vi. It is extraordinary that these are things that don’t have flesh and blood.
JL: Jim would say that every once in a while we were looking at shots. He’d say, “You know, I just have to stop and remind everybody here none of this is real.”
DM: Well, if you guys believe in it, it certainly shows on film.
JL: Yeah, yeah. Cool. Thank you so much. Glad you liked it.
With that, Joe had to run to a meeting at Warner Bros, and I headed back to join my family for more Christmas festivities, so it wasn't until later that I realized there was one question I forgot to ask about one particular effect in the film. Despite all the giant blue cat people and the alien monsters and the spaceships, one of the things that blew my mind was the way Sam Worthington's legs looked when we see him in his wheelchair, and I wanted to ask Joe if those were digital legs or how they were accomplished. I sent him the question through the publicist who set us up in the first place, and he ended up sending me back the following response:
"Those were prosthetic legs made by Richard Taylor and Weta Workshop. Sam's legs were in green and going straight down through the seat of the wheelchair, then painted out. These were shots that ended up being farmed out, so Weta didn't do the final comps on them, but we can probably find one of the plates if Drew wants to see what it looked like on set."
It's an amazing effect and deceptively simple. Thanks to Joe for taking the time and to Carol Marshall for putting us together in the first place.