INTERVIEW: Visual Effects Master Joe Letteri Discusses Creating Worlds, Creatures, and ‘War for The Planet of the Apes’

When it comes to Visual Effects in film, Joe Letteri has worked on some of the biggest and best. The four-time Academy Award winner started his career at Industrial Light and Magic before transitioning to WETA. Working for the two biggest effects companies in the industry, Letteri had the opportunity to design for films like “Jurassic Park,” “The Lord of the Rings,” and “Avatar.”

Letteri also crafted the effects in the “Planet of the Apes” franchise. This year, he is up for his third Oscar for the series. Surprisingly, the films, which have been considered groundbreaking in the world of performance capture, have never won the category. That could change this year with “War for the Planet of the Apes,” the trilogy’s finale.

I recently had the opportunity to talk with Joe Letteri about his work on “Planet of the Apes,” as well as some of his previous projects, and a little bit about what’s next.

Karen Peterson/Awards Circuit: Thank you so much for taking some time to talk to me today. Congratulations on another Academy Award nomination. This makes how many total? Ten?

Joe Letteri: Ten, yeah.

KP: Ten nominations. And you’ve won four. Does it ever start to feel like it’s just another thing or is it always exciting when you get that recognition?

JL: It’s always exciting. You never truly know when you’re working on these films if you’re ever really going to get it or not. You’re always just scrambling to get them done. So it’s nice when it all comes together and you see that “Oh, that actually worked.” And it’s nice when other people appreciate that.

KP: You’ve had the opportunity to work on some of the most iconic films of the last 25 years or so. How have you seen the industry change over the years? Specifically as it relates to what you do.

JL: In the early days when I got started it wasn’t even really clear that people wanted to use computer graphics in filmmaking. There were a couple of early tries with “Tron” and things like that and ILM experimented with getting a few shots here and there like “Into the Abyss.” But, really, it was probably… I would have to say “Terminator 2” felt like the film that really kicked off the industry, really kind of accepting it and looking seriously at it. I was at ILM at the time, but I wasn’t working on the film.

But then the next one that came up for us was “Jurassic Park.” And that’s the one that really felt like it put digital effects on the map, because that was a way to create this idea that you could re-imagine dinosaurs. You could bring them back to life. It wasn’t just fantasy, it wasn’t just science fiction. It was something we were all kind of familiar with, and had a realism to it. And that really seemed to be the film that, after that happened, everyone started thinking about how you could use computer graphics to tell stories.

And, for me, I was really lucky because that carried through into a couple of other films until I met up with Peter Jackson and got to work with him on “Lord of the Rings” to create Gollum. Because, that interest for me—you know, doing the dinosaurs—started sparking my interest in doing characters as well. And Gollum was just a fantastic character. Just, from literature. I loved him as a character in the books, and to get to work with him and to come up with this idea that now you can have a digital character portrayed by an actor who can perform alongside other human characters in the film. For me, that kind of got everything going with the characters we’ve been able to create in a lot of the films I’ve been able to work on since then.

KP: And it’s really astounding the level of detail you’re able to capture. But going back quickly to “Jurassic Park” and “Terminator 2.” Those movies came out when I was in high school, and they were huge and you were seeing things you’ve never seen before. But what’s amazing to me is now, I can still watch them 20-something years later and still be amazed at the level of detail and how realistic they still look.

JL: Well, good, that’s nice to hear. You know, I remember at the time we were thinking we hope this lasts like 10 years. (laughs)

KP: And here we are all these years later.

JL: Yeah, because what we knew even at the time that the tools we had to do what we were trying to do were so crude. And that’s been a lot of the what’s happened since then is the success of those films has allowed us to build more and better tools for creating the even better and more realistic effects. And that’s fed more of the films we’ve been able to work on.

KP: When you moved into motion capture. Which is really fascinating. How did you first get into that and what were some of the early processes of developing it?

JL: Motion capture had been used a couple of times in films before, but it was a bit of a one-off. It wasn’t used consistently to create a lead character in a film. But what happened was, when we were working on Gollum, we had Andy Serkis in the scene playing his part with all the other actors. Originally, he was supposed to just be doing voice off to the side. But because he’s an actor, Peter put him in the shots and that was great. You got the interaction with the other actors, you really got the sense of drama going back and forth. And we had to take those shots and look at Andy. Of course we had to paint him out and put Gollum in there, and we were thinking we’d use his voice and just animate his body using key frame techniques like you would do with any other CG character.

But what we quickly came to realize was, Andy’s performance really was the heart and soul of the character. And was there a way to use that more directly?

We had a motion capture stage, but it was a bit of a science project at the time. What we did was we turned it into a full on production stage. We brought Peter in to direct Andy, and went back through the scenes and recreated what Andy had done while Peter was directing the live action. And we found that that gave such a realism to the performance that we kept carrying on with the idea.

So when we did “King Kong,” we thought, “Well, now we’ve done that with a body. Can we capture that with Andy’s face?” And we came up with a way to do that. Then we were talking to [James] Cameron afterwards about doing “Avatar.” And Jim wanted to do not just one character, but with a whole bunch of characters working in a virtual world. So we thought we’d put a head rig on the actors and that way they can move anywhere they want and we can still capture their faces.

And then we came to do “Planet of the Apes,” we thought, rather than having Andy do this twice, can we figure out a way to do it live while it’s being captured? So we kind of cracked what we needed to do to make that work. So there’s a whole side of it that’s just based on wanting to work with actors and directors and the whole film crew in the moment of making a film. You know? To be a part of that. To not just be at post-production, but to be creating these characters as if they existed in front of us.

Performance Capture
L-r, Karin Konoval, Terry Notary, Andy Serkis and Michael Adamthwaite on the set of Twentieth Century Fox’s “War for the Planet of the Apes.”

KP: It really adds such an extra level of realism. I watched a featurette specifically about “War for the Planet of the Apes” that showed how they took the transformation from Andy’s face to Caesar’s and I was fascinated by that. What were some of the challenges you faced with the “Planet of the Apes” films? Either with the trilogy or with this film specifically?

JL: We talked about the performance capture aspect of it, and that really is what it sounds like. You’re recording the actor’s performance with a bunch of characters and trying to get enough information to understand what the actor was doing and what the actor was intending during the scene. But we still have to translate that into the character you see on the screen. And even though it’s called performance capture, there’s actually a whole different process.

And that started back on “Rise [of the Planet of the Apes].” Because, with “Rise,” we started with the idea that we’re gonna see this evolution from genetic engineering that’s going to affect real chimps. So we had to start with real chimps that looked completely as if we photographed them out in the wild—even though they were digital creations—and knowing we’d have to evolve them to start walking upright, to have dialogue, that this was going to be part of their character arc. And so it was kind of walking that line between realism and naturalism. But then giving a human performance that the audience would understand, because chimps obviously can’t perform. They can’t do dialogue. They have to express themselves a little differently than we do. Even though you’d think they’d be the same because the face is so much the same.

So that translation was something we had to learn, and we were very fortunate to have three films to work that out on. Because, by the time we got to this third film, there was a lot really deep emotion in the performances, and Matt Reeves used a lot of really tight close ups and held on the characters a long time. And that kind of subtlety is something that would be really hard to do with a character unless you really have the time to work with them.

KP: How closely did you work with Andy in developing the character?

JL: Oh, very closely. With Andy, we’re always talking about who is the character? What are you thinking about playing? What are you learning as you play the character? As the whole shoot progresses, what changed from what you thought at the beginning to what you thought he was going to be at the end? All those things that Andy’s thinking about to create the character, we want to know because we have to be faithful to that. It’s not just a mechanical process. You don’t just look at what Andy is doing and think if I just move this little bit of his mouth over to the left a little bit he’ll look angry. It’s not that entirely. It’s also that you have to know what the intention was. Because you have to bring the performance out from the inside.

KP: As a film viewer, I take so many things for granted. It’s amazing how you can make it look effortless, even though there is so much going on behind the scenes. How long does it take to do a film like “War for the Planet of the Apes” on the VFX side?

JL: Matt would involve us even back in the story phase, just so we would start to understand where he was going with it and to think about what he wanted to do with the environment and ideas for new characters and so forth. And then we have to get involved with planning the shoot because it’s a big shoot. So there’s a lot of work that goes into that. But really, the bulk of our work happens after the shoot. It was over a year that we actually had our hands on the shots we were working on. You know, what you were going to see in the theater.

KP: Do you have any shots or scenes that are particular favorites?

JL: There’s a lot that I like, really. I’d have to say, probably one of the nice surprises in this film was the whole scene with Bad Ape. Because he was so unexpected. Not only to us, but to Caesar and the other apes. The idea that, hey, there’s other chimps out there that this is happening to and we’re not alone. That hinted to a whole other world. But it allowed him to have a different personality and coming from a different set of experiences than Caesar and the others. So they find a way to bond and to figure out what they have to do to take on the colonel. And so I thought that whole scene was just really nicely played out and I was glad we were able to capture that.

Steve Zahn stars as Bad Ape in Twentieth Century Fox’s “War for the Planet of the Apes,” also starring Andy Serkis and Woody Harrelson.

KP: Yeah, it was a really fun scene. It opens the whole universe up a lot.

JL: Yeah, exactly.

KP: So next you’re working on more “Avatar” films.

JL: Yes, I am.

KP: So those are really happening! It’s not just rumors!

JL: (laughs) Those are really happening. We’re already into it.

KP: Overall, when you look at the different films you’ve done over your career and the things you’re looking forward to, is there anything you haven’t gotten to try yet that you’re wanting to do?

JL: Not that I’m aware of, because it depends on the story. I never knew we were going to be doing “Avatar” until Jon Landau called me up and said, “Jim has this script. Would you like to read it?” And I said, “Sure. Okay.” And it was the same with “Planet of the Apes.” John Kilkenny at Fox sent me the script and said, “Do you think we can pull this off?” And I said, “Yeah, well, we have to pull this off. This would be a good movie. I want to see this!” So you really don’t know until you know what the story is.

We work on all these techniques. How do you do eyes, or how do you do skin? How do you grow forests? These are things that are just interesting, and we’re trying to solve problems in the background. But you don’t know where you’re going to put your focus and where you really have to focus the artistry until you know what the story asks of you.

KP: How do you develop something like a character from “Avatar” that doesn’t actually exist? Where do you start?

JL: You start with the actors. You always start with the actors and think about wjhat the performance is going to be. And then you marry that with the design. The design for the characters came completely from Jim. He sketched them out and knew what he wanted them to be. And we talked about ways to marry the actor’s performance to the requirements of these nine-foot-tall blue aliens with these cat-like features. And that was fun because we got to work in some of these extra ideas. Like using their ears as a cat would for expressiveness. Or using their tails to counterbalance the way they run the way that a big cat would. They’re obviously very athletic. These are things that go beyond human, but in a way that complements the grace of the characters.

KP: I love the design of them. I think they’re incredible.

JL: Thank you.

KP: I know we have to wrap up. But going back to “Planet of the Apes” to close on, what is something you hope is an enduring legacy when it comes to those films?

JL: To me, what’s really interesting about those films—and I felt this way from the original and I felt this way from any good science fiction. I guess the fact that the original was written by Rod Serling probably helps to make this true. But good science fiction like that really is a way of turning a mirror on ourselves and seeing society in a different way, and giving you the chance to ask questions about it in a way that is kind of removed from our day to day reality but still relevant to it.

And that’s what I really like about this idea that the way these stories are portrayed, yes, humans were reaching beyond their means, but it was for a very real purpose. James Franco’s character was trying to save his father. But, that went awry and suddenly now it’s destroying humanity and apes are gaining intelligence. And the way Caesar is kind of planted as an ape that’s raised by humans and has very human values, he actually becomes the the character that carries the spark of humanity into this new world. So it allows for a whole possibility of storytelling. It just opens up different ways that we can be looking at ourselves and telling a lot of stories with an infinite set of possibilities.

We at Awards Circuit would like to thank Joe Letteri for taking the time to speak about his work. He is nominated for Best Visual Effects for “War for the Planet of the Apes,” along with Daniel BarrettDan Lemmon, and Joel Whist.

What do you think?

AC Fan

Written by Karen M. Peterson

Karen Peterson is a writer from Southern California. When she is not at the ballpark cheering on her LA Angels, she can usually be found in a movie theater or in front of the television. Karen is obsessed with awards shows, and loves everything from the smallest indie film to the biggest of big budget spectacles. She is also unapologetically in love with Tom Cruise.

2017 Oscar Circuit: Best Production Design

WATCH: Anger Management Isn’t Going Well in Latest ‘Marvel’s Jessica Jones’ Trailer, Images