SpongeBob Squarepants Jumps to Life in 3D with Iloura

AEAF ENTRY View in Online Festival

Animation and visual effects studioIlourahas had the privilege and the challenge of re-creating and animating SpongeBob Squarepants and friends as 3D CG characters for his second movie, ‘The Spongebob Movie: Sponge Out of Water’. The team completed 640 shots for the project, comprised of 2D animation augmented with extended sequences of animated CG composited into live action photography.

Iloura’sVFX supervisor Glenn Melenhorst,executive producer Ineke Majoorand a team of 100 artists worked with director Paul Tibbett and SpongeBob creator Stephen Hillenburg to make the transition of the SpongeBob characters from 2D to 3D as authentic as possible. After 15 years of life on screen as 2D characters, gathering a huge fan base of adults and children, it was essential to retain their aesthetic and personality.

SpongeBob and Friends

The team’s R&D and character design period came mostly during the time Iloura spent preparing their pitch for the project, which was completed over a two or three month period. In that time, they ran movement and distortion tests and worked out how they would need to modify the team’s regular pipeline to deal with the SpongeBob characters.

Eventually they developed a series of approaches for re-creating the characters in a 3D environment, in a manner that was sympathetic to the existing 2D animation and immediately recognisable to audiences. Once Iloura was awarded the job, the team spent almost two years on the project from the first meetings to delivery of the final shots.

Somewhat over half of the film is purely 2D animation, and precedes the part that involves the 3D characters. The team was very aware of the visual leap the audience would have to make to follow the story as the familiar characters venture into the live action, 3D world. As they worked, the team was in contact with the creators and writers of the original TV series, Stephen Hillenburg and Glenn Berger, almost daily via cineSync to keep a firm vision in mind of how the characters should look and behave.

VFX supervisor Glenn Melenhorst said, “The original animators had never thought about these characters in a 3D environment, and their distinctive shapes had never intended to be viewed from all angles. Therefore, getting SpongeBob and his friends to perform convincingly and look authentic on screen was a special challenge. Even the static toys created by merchandising companies over the years were not especially helpful, as they didn’t need to move, communicate or show personality.”

Animation Style

Realising that the client wasn’t expecting SpongeBob to look like a photoreal kitchen sponge, or his starfish friend Patrick to match a true-to-life creature, they began with a set of sculpts in ZBrush. Meanwhile,lead modeller Samuel Jensenproduced macquettes to help the artists visualise the 3D models as they built, rigged and animated the SpongeBob cast.

After exporting the initial sculpts from ZBrush, the characters were modelled in 3ds Max or Maya, rigged and animated in Maya and rendered in 3ds Max with V-Ray. In terms of detail, instead of photorealism, they focussed their efforts on details that would evoke the original characters. They also made the decision that in mid-ground or wide shots the models would be relatively smooth, and reveal more texture and small details whenever the camera moved up close. Nuke was used for compositing.

Between them, the artists at Iloura also had years of collective knowledge of SpongeBob, either as children themselves or as parents. This was helpful when it came to the animation, since there was no list of rules from the client about what was going to make the characters work. It was a matter of watching many hours of video and instinctively knowing when it was right.

Understanding the motivations for most actions was fortunately fairly simple, but from a staging point of view, posing and creating an effective silhouette was more challenging. The over-the-top stretches and contortions of a 2D drawing cannot always be replicated by a 3D model composed of polygons, of which only a limited number can be pushed and pulled around.

Post-Sculpting

Iloura was especially keen to avoid turning down requests from Stephen Hillenburg and the movie director Paul Tibbitt about the animation. Instead, they worked on building flexible rigs.Lead CG supervisor Avi Goodman’s rigs allowed them to break the characters to a large extent during performances. Furthermore, whenever they encountered moves that traditional CG could not accommodate, they used a customised post-sculpting process to correct any mistakes caused by pushing the rig too far, or simply to enhance the strength of a pose.

Glenn described their technique. “For example, Patrick might have his arms hanging down by his side and forming a smooth shape, but as soon as he lifted them up, the rig would bulge in weird, unexpected spots and wouldn’t allow him to reach the desired pose. So, once the overall move was signed off, the post-sculpt team would locate every keyframe that needed correction and sculpt the shape into a nice clean line in 3D, by winding the corrections in and out as blend shapes on top of the animation. They could also adjust the facial animation to match the distortions. Just about every animation shot needed post-sculpting.”

Pulling Faces

For the facial animation, the team did not work directly with voice actors but based their moves on audio clips. The characters’ faces look deceptively simple. “Even SpongeBob’s face is not that easy to animate. Viewed straight on in 2D, his distinctive round cheeks may or may not be visible, depending on the shot. Also, as he turns sideways the mouth suddenly becomes a sharper triangle shape,” said Glenn.

“But these techniques won’t work in 3D – making the cheeks, freckles and so on pop in and out of view simply looks creepy in 3D. We have to turn his head more deliberately and let the cheeks emerge gradually. Even with the lip sync we needed to introduce a few 3D tricks. If you really pull the mouth around and make it a perfect shape from the side, it casts odd shadows onto the head. It all added up to a lot of shot by shot tweaking.”

When they eventually wash up on the beach at a seaside town and venture into live action sequences, the SpongeBob characters are quite small, only about 15cm high, which meant bringing the live action camera down close to ground level. This, in turn, introduced depth of field issues when the animators needed to show the whole group standing together and keep them in focus within the plate.

They also needed to consider the physics of being this size in the real world as well, such how large and heavy the grains of sand should be in the particle systems they had to create for interactions, or what would happen to ice cream they interacted with. Simulations for this type of effect were created in Houdini or 3ds Max, and rendered in Max.

Super-sized

Eventually, the story takes a super hero twist when the friends scale up into full-sized muscular characters that needed a second, complete set of 3D animated models and rigs. “In a few instances like SpongeBob’s facial features, we could apply what we had learned from designing the smaller characters,” Glenn said. “But overall these super-sized versions were completely new. For example, the character Squidward’s leg count fell from four to only two. Proportions, shapes and moves were all different and required separate rounds of character development, turntables, macquettes and approvals. They even needed new super-heroic outfits - lycra jumpsuits with custom stitching – which added the complexity of character-cloth interactions during animation.”

Most of the seaside live action shoot took place under bright sunshine in Savannah, Georgia. Iloura was on set throughout the photography. They knew that integration of the little CG characters into the environment was critical to their success and would take some extra care and attention to detail. Almost every set had to be recreated accurately in 3D in order to place, animate and integrate the CG realistically into the plates, and for that reason LIDAR scans and extensive photography were captured at the shoot locations.

Creating each of those environments as a digitised reference not only assisted integration during compositing but also helped set up 3D cameras, lighting rigs and reflections. For example, the entire city street scene could be reflected back onto Mr Krab’s shiny body and onto eyeballs, and shadows from irregular elements like trees could fall accurately onto the characters. Lighting was an ongoing challenge for many of the live action shots, in fact, sometimes handled by replacing the skies, relighting their CG elements, or using roto shapes to help CG elements interact more convincingly with the sharp shadows existing in the plate.

Stand-in props were used on set as further help to bridge that gap between the live-action and 3D CG worlds. While SpongeBob and his friends were still at their small size, tiny macquettes were placed on set to allow the crew to rehearse shots, using plastic rods to puppeteer them around. Then, after removing the macquettes, the camera moves were recreated for the shoot. Having both versions to refer to in post gave the artists many of the clues necessary to refine lighting, focus and intent of animation. Much larger cut-out props were placed on set to represent the superhero versions, and were vital to set up and frame the shots effectively.

Seagulls & Skeletons

Iloura’s 3D work for ‘The SpongeBob Movie: Sponge Out of Water’ went beyond SpongeBob and his friends. The flock of seagulls that join Burger Beard on his pirate ship started with a largely photoreal concept, except for a few concessions that made them more expressive and anthropomorphic, such as the design of their eyeballs. Their eyes are larger than real gulls’ eyes and more forward-facing to allow them to work better for their comical songs and the conversations they have with Burger Beard.

Otherwise all body mechanics, feathers, wing animations and so on maintained a photoreal approach. The team were almost pedantic about their flight, landings and behaviour on the ground, and kept their beaks intact after finding that putting mouths on them to talk made them too cartoony.

The skeleton pirate was a late addition to the story, based on a re-shoot aiming to introduce more clearly the essential, enchanted book in which people can write the future, ahead of time. The skeleton starts as a practical prop at first, sitting on a throne on the set. A 3D asset was built and keyframed, based on this prop’s costume, skull and bones. From the moment in the story that Burger Beard snatches the magic book from his bony hands, the skeleton pirate is replaced in the plates by the CG model.

Pirate Ship

The pirate ship was variously a live action or a partly or completely CG asset. Its sails, mast and ship’s rigging were digital in every case. When we see Burger Beard at sea with the seagulls, the actor was shot in a physical ship on a green screen stage in a studio. Iloura replaced the backgrounds with plates they shot in Hawaii. When we see it driving through town, it is a driveable prop.

On the other hand, when the ship sails up out of the water onto the beach, and when it ultimately crumbles apart in the car park, digital assets were used. Interestingly, although the day of the shoot for the ship scenes taking place at the Georgia beach town was cloudy, for continuity they used their scans and photography to re-build the town, track in the ship and re-light the complete scene with lighting information from the sunny days.

ThePelican Islandsequence in the story was a special effort from the CG team. They decided that everything, from the island’s rocky outcrop with waves crashing round it, to the pelicans swooping around overhead, would look best if the shots were entirely digital. The ocean was the primary challenge. Iloura’s team have been using Naiad for water simulations for several years, since before it was acquired by Autodesk, and used it very effectively in the movie not only for this island sequence but for most of the water interactions in the green screen and beach town sequences as well. www.iloura.com.au