Pixar’s RenderMan Meets Kosmo: Bringing the Real World Into the Creative Pipeline
From Toy Story to Jurassic Park, one name has been behind almost every major visual effects production in Hollywood for more than three decades: RenderMan. Pixar’s industrial-strength renderer creates the final images for world-class productions such as The Mandalorian. Along the way, it has played a role in numerous Academy Award-winning visual effects productions, consistently setting the industry standard for film and television rendering.
Now, RenderMan has a new creative partner. It is experimenting with real-world data captured using Kosmo, a 3D world data engine developed by Hesai.
After receiving a Kosmo spatial camera, Dylan Sisson, Head of RenderMan Creative Engagement at Pixar, set out to explore what happens when real-world environments meet professional rendering tools. In a series of striking experiments, he scanned the plaza at Pixar’s headquarters in less than 20 minutes, imported the data into Houdini via Universal Scene Description (USD), and rendered it directly with RenderMan. Instead of painstakingly rebuilding the environment with traditional polygonal geometry, he transformed the plaza into an impressionistic scene resembling a living oil painting. He went on to bring Pixar’s experimental studio, The Lab, into the digital world in its entirety, capturing a handmade cardboard television frame in such fine detail that every trace of glue was recreated.
These experiments offer an early look at what is possible. Sisson demonstrated that creators can control the stylization of real-world 3D Gaussian Splatting (3DGS) data with film-quality precision, all within a professional film and television rendering pipeline. They point to a future in which RenderMan renders not only virtual worlds but also “a fully digitized real world.” The “magic wand” bridging the physical and digital worlds is Kosmo.
RenderMan: The Engine Behind Hollywood’s Visual Effects
RenderMan is Pixar’s core rendering technology, with foundational algorithms that predate the company itself. The product officially launched in 1989, six years before Toy Story. Pixar’s RenderMan was used by 24 of the last 30 winners of the Academy Award for Best Visual Effects, including every winner for 17 consecutive years from 1996 through 2012.
It all began with the conviction of three founders. Ed Catmull believed computer graphics could be an art form. Steve Jobs saw the potential and made a major bet on it. John Lasseter dared to imagine the future of animation. Together, they not only founded Pixar but also built RenderMan, a tool that turns technological innovation into creative possibility.
From The Abyss to Jurassic Park, RenderMan was behind the images that left audiences asking, “How is this possible?”, including the first lifelike dinosaurs on screen. It has produced the final images for world-class screen productions including Toy Story and The Mandalorian. It renders materials such as metal, skin, or flowing water in ways that are virtually indistinguishable from reality. From large-scale productions involving billions of pixels in a single film to precise control over visual style, RenderMan has consistently represented the industry’s highest standard.
Today, RenderMan continues to evolve. Its GPU-accelerated XPU architecture is in place, and it now supports 3DGS, a new 3D data format. Sisson’s experiments with Kosmo explore how that support can bring captured real-world environments into RenderMan.
Sisson in conversation with Hesai CEO David Li in San Francisco
Dylan Sisson, Head of RenderMan Creative Engagement at Pixar, joined the company in 1999 and jokes that he is “still the new guy at Pixar.” His role is a distinctive one: he follows cutting-edge developments in rendering technology while understanding what new software capabilities mean for the global creative community. He acts as both a “translator” and a “bridge” between Pixar’s technology and the creative world. He is also an active artist in his own right. Recently, Sisson sat down with Hesai Technology CEO David Li in San Francisco for an in-depth conversation covering Pixar’s three decades of technological development and the possibilities Kosmo could bring to film and television.
How a “Magic Wand” Entered Pixar’s Workflow
The story began in Hong Kong. During SIGGRAPH Asia 2025, Sisson accepted an invitation from Hesai’s Kosmo team to try a spatial camera still in development: a device that could “transform” the physical world into digital assets. He described it as a “magic wand” for capturing the real world. A few months later, the product became a reality, and Sisson received a Kosmo spatial camera.
His first experiment focused on Pixar’s own plaza, home to the famous Pixar Lamp and Pixar Ball in front of the Steve Jobs Building. Capturing the space with Kosmo took just 10–20 minutes. The resulting high-quality 3DGS data was imported directly into Houdini’s Solaris environment in USD format, then rendered directly with RenderMan.
Pixar’s plaza, scanned with Kosmo and rendered in RenderMan. Source: Sisson’s blog (2)
Sisson summed up his experimental workflow in a simple formula: Reality → Splat → Render.
From there, he began exploring the artistic possibilities. In Houdini, he manipulated the captured data as points, adjusting their positions, scale, and opacity. He took the experiment further by transforming the entire plaza into a painting with flowing brushstrokes. Using camera-based falloff controls, he kept the ball sharp while allowing the background to become increasingly abstract. He also used AI agents to drive Houdini, guided by a design document specifying effects such as painterly foliage, objects fading into the distance, and brushstrokes that followed the underlying geometry. He then reviewed each render, adding paint-over work and refining the final image.
Sisson scans a cardboard TV frame with Kosmo
Pixar’s internal experimental team, The Lab, also asked him to scan a handmade cardboard television frame. The scan captured its folded edges, corrugated texture, and handmade knobs in remarkable detail, down to “even the glue on the cardboard.” He went on to scan and stylize the entire Lab, capturing art supplies, tools, toys, and even a Jaws pinball machine. He described the result as “a highly detailed digital model and a time capsule.”
Gaussian splats allow creators to capture the real world in photorealistic detail and turn that data into flexible creative assets. For Sisson, Kosmo has opened up new ways of working: “With support for splats in RenderMan, in Houdini and Solaris, and in OpenUSD, plus the ability to create splats with Kosmo, I’m suddenly able to unlock new workflows I wouldn’t have done otherwise. It’s not a matter of being faster — it’s a matter of unlocking new creative possibilities. It’s beyond time and money. That’s really what Pixar is about. It’s not about making movies faster. At the end of the day, it’s about what we can do creatively and what we can envision and bring to the screen that we couldn’t before.”
Kosmo’s photorealistic detail captures the real world, giving artists the creative freedom to manipulate individual objects, as shown here
To describe the value that matters most in film and television creation, he used a technical term: directability. “The director has to be able to say ‘I want to change this,’ and it has to be done at a pixel level — we need complete control. The more data and information we have, the more nuanced the adjustments we can make. Being able to take a Gaussian splat into an environment is becoming more and more common in visual effects. So having a tool that lets us go into a place and capture the spatial elements — the geometry, the lighting — is really interesting and timely.”
He also sees longer-term significance: “Look at what ILM has done with StageCraft and volumetric LED capture. When that gets democratized — when mid-tier studios and even schools start using this technology — it gives people the ability to tell stories they wouldn’t have the capacity to tell otherwise. And Kosmo is one of the steps in making the ability to create splats — and get high-quality content into visual effects — available to more people.”
On October 5, Pixar released the official tutorial “Gaussian Splatting the Pixar Campus” on the RenderMan website, showing creators around the world how to work with real-world 3DGS data in Houdini and RenderMan (3). The demonstration scene is the plaza at Pixar’s headquarters that Sisson scanned with Kosmo: more than 1.25 million Gaussian splats faithfully recreate the lamp, ball, and vegetation in front of the Steve Jobs Building, down to the sunlight and shadows.
In the tutorial description, Pixar states: “We scanned this real-world scene with a Kosmo spatial camera. Thanks to Kosmo for providing the hardware used to create this scan.” On the website, Pixar describes Kosmo as a spatial camera that captures real-world locations and objects as detailed 3DGS data for quick integration into 3D creative workflows.
More importantly, Pixar has shared the experiment in its entirety. The complete 120 MB project file is publicly available for download, along with conversion scripts that allow creators to directly substitute their own scan data. With the invitation “Go Splat Yourself,” the tutorial encourages creators to scan their own world and try the workflow themselves.
The tutorial gives creators a practical starting point for exploring Kosmo data with RenderMan. Using the shared dataset and project files, they can examine Sisson’s experiment and explore how a similar workflow might support their own creative work.
From One-Time Capture to Ongoing Creation
Sisson’s experiments illustrate potential applications for film and television production. Traditional film and television production records a single, fixed viewpoint: footage is limited to what camera captured on the day. A new angle or a change to the scene can require returning to the location for another shoot. Even before filming begins, location scouting can involve dozens of people working for weeks, while building a physical set can take months.
With Kosmo, filmmakers can capture a location once and work with it throughout production. In preproduction, a single operator can digitally capture a scene in one day for previsualization, potentially reducing location-scouting costs from around 30% of the preproduction budget to 5%. The captured environment can then be integrated into established tools such as Unreal Engine and used during filming in LED virtual production or green-screen workflows. A 10,000-square-meter European street scene, for example, would take two months to build physically; scanning, reconstruction, and editing a digital version can take just one week, with cost savings of up to 80%. In postproduction, individual elements, down to blades of grass and trees, can be edited and reused as production-ready 3D assets. Artists can turn day into night, adjust the lighting, and create additional shots from the same captured environment. Real spaces are no longer “one-time filming content,” but fully digitized 3D assets that can be preserved permanently and reused repeatedly.
Reflecting on a tree he scanned on a San Francisco street, Sisson said: “This tree is like this today — and I get to capture that.”
Sisson’s experiments demonstrate how real-world spaces captured with Kosmo can become high-fidelity, flexible digital assets that artists can edit, stylize, and reuse within a workflow built around Houdini and RenderMan. By combining faithful capture with precise artistic control, these demonstrations highlight new ways to bring the real world into film and television production. Kosmo is opening new creative possibilities—giving artists the freedom to capture the world as it is and reimagine it as it could be.
（1）Universal Scene Description (USD)，a 3D scene format developed and open-sourced by Pixar, is now widely used as a data interchange standard across the film and visual effects industry.
（2）A Brush with Reality. https://www.dylansisson.com/stories/kosmo/ an article on Dylan Sisson’s personal website exploring how to use Kosmo 3DGS data in RenderMan.
（3）Gaussian Splatting the Pixar Campus, an official Pixar RenderMan tutorial featuring scan data captured with Kosmo.https://renderman.pixar.com/gaussian-splatting-the-pixar-campus