Needless to say, Peter Jackson’s return to J.R.R. Tolkien’s Middle-earth with The Hobbit: An Unexpected Journey will probably get a lot of attention for its groundbreaking special effects, mainly since all three of Jackson’s previous “Lord of the Rings” movies won the Oscar in that category.
At the head of the visual FX department creating Jackson’s fantastical world and characters is Joe Letteri, who returns after doing visual FX on The Lord of the Rings: The Two Towers and “The Return of the King,” which he followed up with equally-respected FX films like Jackson’s King Kong and James Cameron’s Avatar (for which Letteri won two more Oscars) and last year’s Rise of the Planet of the Apes.
ComingSoon.net got on the phone with Letteri last week, a nice counterpoint to our earlier interview with Andy Serkis, since the two worked hand-in-hand on the performance capture aspects of the film, which we discussed among other things such as Jackson’s decision to shoot at a higher frame rate. In other words, for those of you who like your interviews to be very tech-heavy, this is the interview for you, although Letteri gracefully side-stepped when we tried to ask him about his next big movie, Zack Snyder’s Man of Steel.
ComingSoon.net: Since “The Lord of the Rings,” you’ve done a lot of groundbreaking FX work – “King Kong,” “Avatar” and the “Apes” movie. When Peter got you on board for this new one, was there anything that struck you that might be a big challenge or had you worked that all out since the previous movie?
Joe Letteri: It was both. We had a lot of techniques going that we always keep progressing forward, new advancements in how you do eyes, how you make skin look like skin, hair, cloth, how you build a mountain, how you build a tree, all those things that have now just become part of the toolkit that we need because we obviously could be asked to create anything. Having said that, there’s still a lot of work that goes into making this. You take a scene like the goblin cavern and you have to build this big three-dimensional landscape and anchor it with places people can walk around in and do all this action it’s not that much different than building a live action set. Even if you have the tools to do it, it’s still not the push of a button. There’s still a lot that has to be crafted by hand and all put into position and lit and choreographed to make it all work. It’s a bit of both, but we knew going into it that we wanted it to be big. We wanted it to be like we were returning to Middle-earth in the best possible way.
CS: I spoke to Andy Serkis earlier and I asked him about the goblin cavern scene specifically because I thought that might be hundreds of extras running around, but a lot of it was constructed based on what they did on set.
Letteri: Yeah, we did some small pieces of the set that the guys could run around on and do some of the action especially that bit where they’re brought before the Goblin King because a bit of the action happens on that well-lit platform in front of the throne but most of what you see in there is digital. In fact, a lot of what was shot, we wound up replacing, just because of the complexity of the camera moves and the stereo. We shot goblins with partial suits on, but we were going to be adding their heads. A lot of what we were really doing that for was for a couple shots where it was practical. Most of the time it was easier to just replace them with completely digital characters then to try to match heads onto partial bodies. Sometimes we ended up just replacing everything that shot including the actors, because it was just easier to construct the big shots that way. Yeah, we’ve gotten to the point where we can make characters and digital doubles. As long as you’re not talking about nuanced emotional dialogue performances, you’re talking about action scenes, sometimes it’s just easier to go all digital with it.
CS: I also wondered about the dwarves. In the previous movies, you had a way of turning full size actors into hobbits and you used some on-set trickery to do that, but here, you were shooting 3D and you had 12 dwarves played by 12 actors of different sizes. So wouldn’t that make every shot an FX shot because you had to make them look dwarfish?
Letteri: Yeah, for the most part. There were a couple of shots where you could just get away with putting Gandalf up on an apple box or having the dwarves down on their knees. As long as you maintained a fairly correct perspective spatially because stereo obviously picked up the difference, you can get away with that, but when you go wide or when the camera started moving, that’s when you really needed to go in and look at either rotoscoping Gandalf off of the plate and you scale him up and lock him back in 3D or you replace him with a 3D character all together, like a digital double, or you go with this dynamic forced perspective using synchronized motion control rigs.
CS: Was that something brand-new you used for this movie or was that something that’s been used before?
Letteri: It’s new as far as I know. The idea has evolved for us out of something we were doing on “Avatar” where we were doing motion capture that we were then feeding into the live action cameras so that Jim and the operators could actually frame up on the action so you’re not just shooting tennis balls when you’re doing your camera framing. We pre-recorded that action on “Avatar.” On “The Hobbit” what we did was for the virtual characters like the trolls, we used the same technology. We had a performance capture stage set up next to the live action stage with a virtual camera that was being driven by the live action camera and then the troll animation, the actors that were performing live was being fed into the monitors so the operators could see it and Peter could see it and Peter could direct and choreograph everything all together. When you needed to do two live action elements, this was kind of like the ultimate extension of all of that put together. You have the camera on let’s say the door in the Bag End set but now rather than driving a virtual camera, it’s now driving another 3D stereo rig but this one is not a motion control rig so it’s being slaved to the main camera and then the actors in this case such as Sir Ian would be acting on a green screen stage because he was real, not virtual, and they’re having to choreograph their moves so they don’t go through each other in 3D, which is something you can’t see on the monitors now because you’re compositing the images live and through this elaborate rehearsal and choreography and getting all the cameras to synchronize and fire correctly. Eventually, you can get all the pieces together so that we can take everything and composite it into a shot. It still requires a lot of nudging things around spatially, like element by element in 3D, to really get it to lock down but at least you’re in the right place to begin with.
CS: The other big factor was the fact Peter shot in 48 frames per second, so is all the software your animators use capable of doing 48?
Letteri: It wasn’t two years ago but by the time we started, we made sure that it was. We talked with everyone that needed to be involved, both for in-house software and anything we were working with externally, like talking to Foundry, to let everyone know this was coming, just like we did with stereo when we did “Avatar.” When we started rigging everything up, we just started prototyping it and showing them what we did and we got them to add that into the released versions of the software, so the same thing happened here. Again, we were ready to go when the production started.
CS: What’s the general reaction to 48 frames per second among the FX people you work with? Does that seem to be something that will continue into the future and does it help enhance the FX and 3D of it?
Letteri: Well, it solves one problem in 3D, which is motion blur. If you tend to be prone towards motion sickness when you see something really blurry in 3D, this really alleviates that problem, but it does change the look of the film because everything looks hyper-real. Yeah, as Peter mentioned, and I’m hearing the same thing too from the few screenings I’ve been to, when you talk to people, the more familiar you are with film and the more nostalgic you are for film, the harder it is to let go, and after a while, you can’t let go, and you take it for what it is. Anyone who I talked to who is younger who is used to seeing film in any number of different ways says it’s like really not even noticeable, not really a big deal.
CS: It’s interesting because you’ve been doing visual FX and for at least the last twenty years, it’s mostly been done in the digital domain, so I was curious about where your department stands on film vs. digital.
Letteri: I’m one of the people that’s still nostalgic for film. I like the look of film. For me, it’s still a really great look, but on the other hand, going digital gives you some real advantages that have been able to surpass film, like what you can now get with contrast and dynamic range and lack of grain and resolution and steadiness of the camera. Without digital, you couldn’t really practically be doing 3D so now if you’re going to shoot in 3D, you pretty much have to do digital and now you’re fully into this realm. Okay, now if you want to solve some of the motion artifact problems of 3D, you have to go higher frame rate, so these things kind of drive themselves. They’re all pointing to the fact that if you’re going to make an experience more immersive, you just have to go more and more realistic with it.
CS: Interesting. You talked a little about performance capture and its evolution and you’ve been there every step of the way from “Lord of the Rings” to “King Kong” to “Avatar” and “Apes.” Was there another step in the evolution for this movie besides what you mentioned earlier?
Letteri: I think the evolution as far as the motion capture was sort of what described it, taking us all the way back to the digital out of it in a way and you just went back to the live action but you still couldn’t be thinking about doing those two slave camera systems I think without having gone through the whole experience of doing it digitally. Not that it couldn’t be done and not that you couldn’t think about it, because I know people have talked about experimenting with this idea before, but I think the practical production experience we gained of going through this step by step on these different films made it tractable to just say, “Okay, this is not really a huge leap from what we did last time, this is just the next step.” So there was less fear involved for everybody to take that on board and try to shoot the film that way.
CS: You obviously have two more movies to finish in this trilogy and I guess you’re also working on “Man of Steel” and I know James Cameron wants to do more “Avatar” movies and there’ll be another “Apes” movie. Are you grooming others to take over those projects? How are you managing all of that?
Letteri: I sort of operate at overseeing everything, but every film has its own team now, visual FX supervisor, producer, so even on this film, Eric Saindon was our visual FX supervisor and he was on set every day with Peter. We had a 2nd Unit visual FX supervisor, Kevin Smith, he was with Andy, so really, I was only there on set for a few weeks of prep and a few weeks of the shoot getting started. Once everything was up and running, I was back and working on other things. In fact, mostly I was finishing up “Tintin” at the time and then once that was done then after that was done, it was just trying to get everything going for the digital aspect of what was happening. It’s kind of a team effort and it works that way on every show that we do.
CS: Have you had any conversations with James Cameron yet on whether he might shoot on 48 as well? Has he seen “The Hobbit” yet?
Letteri: Jim’s still thinking 60, because it’s a persistence of vision artifacts and to quote Doug Trumbull and the studies he’s done, he says it tends to go away at around 64. I tend to agree with him from the tests we’ve done. We’ve gone to pretty high frame rates and after about 64, the returns are pretty minimal and to tell you the truth, the difference between 48 and 60 is not as noticeable as the difference between 24 and 48 so for practical reasons, we decided to go with 48 because asking the theaters to have projectors and servers that could handle 60 was not really practical in the time frame we had for “Hobbit” plus it would have meant even more work, double the amount of work, that we were already committed to at 48 frames.
CS: You also have two more years and movies for people to get used to it and it to become more available projection-wise.
Letteri: You have a choice. My recommendation would be that if you’re not certain that this is something you really want to see, see it in 24 frames, and if you want to see the film again, which I hope you do, then give it a try at 48, so that way you get the whole experience. Honestly, people have told me that seeing the 48-frame version of the movie, even though it might have taken them a few minutes to get used to it, when they went back and saw the 24 frame version, it felt like a step down. Even though initially they had the experience of “This is what I’m used to seeing” but I would say that if you’re curious, maybe go the other way, maybe go 24 and then 48.
CS: I might have to see it three or four times in every possible permutation to figure out which one I like.
Letteri: It’s interesting because I still like going back to
because we have to watch every version of it as well before it goes out the door, and it’s really interesting still looking at the film version of it and what’s interesting is that–as nostalgic as I am for film and a lot of people are–after you see the digital and you go back to look at the film, you realize how flickery it is and how jittery the projection is, stuff that when you watch film for 20 or 30 years, it looks fine, you never think about it. As soon as you compare it to what it could be, suddenly the deficiencies become glaringly obvious and I suspect that’s what people are seeing when they go from 48 to 24. Sure, it’s a different look, sure you’re seeing more detail then you may have expected going into it, but you’re seeing it like it’s happening.
CS: What do you think is the next step in terms of visual FX? I know you’re working on “Man of Steel” so is the flying still something you have to master or are there other things that you have to deal with that are different from other things you’ve done?
Letteri: There’s probably not a lot I can talk about yet because we’re almost done but obviously the movie’s not coming out for a while, but like all these things, we’re getting to the point where
well, you look at “Hobbit,” you could have taken any one character, say the Goblin King, and if that’s all we did for the film or just the goblin caverns, that would be a fairly significant piece of work, but now we’re actually doing so many things across so many different types of sequences, that to go into a film like “Man of Steel” or any film, we now have a repertoire of techniques that we bring to any film, so it becomes less and less about is there a specific technology, because usually it’s a mix of all these technologies at various stages of advancement. It just becomes more about how do you pull it together? What’s the creative vision? What’s the artistic vision?
The Hobbit: An Unexpected Journey opens in 2D, 3D and 3D IMAX theaters on Friday, December 14.
(Photo Credit: Getty Images)
