Disruption is one of the few constants in the history of cinema. Early film theorists considered silent cinema a universal language until “talkies” transformed storytelling for the big screen. The domestic proliferation of TV after World War II drew audiences away from theaters and in the process helped dismantle the old studio system. The “digital revolution” has recalibrated where, when, and how we watch, even as it has caused cinephiles to mourn the loss of celluloid. Joining this line of creative destruction, the rise of AI-enabled (or “synthetic”) media suggests that cinema is, once again, under threat.
Synthetic media rely on machine learning tools that perform a range of tasks. They enable micro-targeted systems of distribution; streamline production and post-production workflows; and animate, edit, or even create entire audiovisual works from a text prompt.
Many of the hazards of synthetic media, such as malicious deepfakes, have been well documented and need further attention. Still, AI elicits a special kind of anxiety for the film and TV industry’s creative classes. The question now on everyone’s mind is whether feature-length films made by text-to-video generators will eliminate the skilled labor of screenwriters, graphic artists, editors, and directors.
It is doubtful that Hollywood studios will launch a major lineup of AI-generated features anytime soon. Nor will viewers likely want to completely forego the experience of collectively shared films or series in favor of bespoke entertainment they create with a few sentences of prompt engineering. Even as text-to-video software continues to improve at an extraordinary rate, it will never replace the social elements crucial to the product Hollywood makes and the culture that surrounds both gaudy blockbusters and gritty dramas alike. To produce a film is to give shape to the multiple human narratives involved in its fashioning. People—the deals they make, the complex biographies they bring to the script, their chemistry or friction on set, the aesthetic influences that inform their approach to sound design or cinematography—make a picture what it is. (A more pressing concern is that studios will use algorithm-driven predictive analytics to greenlight only those projects they believe are sure to make money, leading to less diversity of form, story, and talent.)
At the same time, filmmakers are gravitating to AI to expand their craft and create new forms of moving image art.
AI aids in the speed and precision of workflows in ways that are important, but not always visible to spectators. Machine learning tools can organize footage so editors can more easily recall particular camera angles and scenes of dialog. And on the other side of a project’s life, restoration efforts involving algorithms remove dirt and scratches from old prints and repair the ravages of warp and flicker.
One of the most prominent ways AI has surfaced on screen has been through VFX work, to shape complex performances. The Wētā FX software Massive has helped effects artists capture the seemingly “unfilmable,” especially on the macroscale. Beginning with the creation of digital hordes of orcs and humans for the realistic combat in The Lord of the Rings: The Two Towers' (2002) Battle of Helm’s Deep, Massive has since been responsible for expansive collections of lifelike entities, from the shiver of sharks in The Meg (2018) to the swarms of flying demons in Shang-Chi and the Legend of the Ten Rings (2021).
On the more granular level of character design, AI has broadened the possibilities for facial and bodily expression. To craft the otherworldly villain Thanos in Avengers: Infinity War (2018), the James Cameron-founded Digital Domain subjected a motion capture rendering of Josh Brolin’s face to an algorithm trained on the actor’s live-action movements. A tremendous improvement on the rigid cartoonishness we glimpse in The Avengers’ (2012) post-credit sequence, Thanos bore all the signs of Brolin’s acting, his skillful communication of anger and sadness so integral to the character. Similarly, AI-enabled “de-aging” has in some cases contributed to the greater continuity of performance, facilitating a bond between actor and character across narrative time. Films as disparate as gangster drama The Irishman (2019), sci-fi thriller Gemini Man (2019), and upcoming action-adventure Indiana Jones and the Dial of Destiny (2023) have used algorithms to digitally sculpt younger versions of the aging lead actors.
A remarkable if ghoulish extension of this work can be seen in “synthetic resurrection,” where a deceased actor or historical figure is brought back to life to play a role in the present. It’s likely that we’ll see deceased figures come back around for the purpose of an eye-catching cameo, a supporting part, or to smooth over the narrative complexities of a multi-year series or franchise. It would be surprising if AI-as-lead became the norm, however. Even if it becomes easier to negotiate “likeness rights,” these simulations will still plunge us into new terrain within the uncanny valley. Viewers’ discomfort would not necessarily stem from the image itself, but from the cognitive dissonance of trying to square the knowledge of the character’s digital construction with how real they seem on screen.
Synthetic resurrection has been particularly contentious in the realm of documentary. While the field positions itself against media deception and manipulation, filmmakers have begun to use AI in subtle ways. One example is the vocal synthesis employed in The Andy Warhol Diaries (2022), an otherwise formally conventional nonfiction portrait of the artist. Director Andrew Rossi, in collaboration with Resemble AI, has Warhol narrating his ruminations to assistant and friend Pat Hackett via an intermittent voice-over. To avoid the controversy surrounding Morgan Neville’s simulation of Anthony Bourdain’s voice for his 2021 biodoc, Roadrunner, Rossi secured permission from the Warhol Foundation and alerted viewers to the use of the technology in a disclaimer. The result is a redoubled emphasis on traditionally “Warholian” preoccupations, encouraging viewers to reflect on the nature of mediation, the fabrication of celebrity, and what can be simultaneously revealed and concealed by public forms of address.
AI has also become increasingly important in human rights documentaries. For Welcome to Chechnya (2020), a film about the oppression of LGBTQ activists there, VFX supervisor Ryan Laney drew on deep learning tools to craft digital faces for the film’s subjects. Laney used volunteer activists in New York to create composite faces that protected the subjects’ identities while still allowing viewers to emotionally connect with them on screen. Laney expanded on these techniques in Faceless (2022), Jennifer Ngo’s vérité-style documentary about young activists on the front lines of the 2019 pro-democracy protests in Hong Kong. Here, Laney scrambled the relationship of measurements between specific points on the protagonists’ faces to stymie attempts to identify them using biometric scans.
Some of the most incendiary works of AI-enabled film, however, exist outside both commercial and nonprofit spheres. The wide breadth of playful mashups and advocacy-oriented PSAs that populate social media and video-sharing platforms serve as R&D for the film and TV industry (and at times offer studios fresh talent), as well as advancing trenchant social critiques. Deepfake satire, in particular, constitutes a form of high-concept culture jamming, mocking duplicitous tech entrepreneurs and authoritarian leaders.
The proliferation of “generative” text-to-image and text-to-video software is pushing AI into a new phase. Instead of looking at the visuals from models like DALL-E 2, Midjourney, and Stable Diffusion as fixed and isolated objects, we could consider how these tools might operate as part of a larger creative process.
Synthetic imagery is sure to become central to preproduction. Screenwriters will be able to use AI-generated imagery for their pitch decks to evocatively establish the mood and feel of a project and position it within a larger genre. Likewise, concept artists will benefit from the back-and-forth tweaking of prompts and visual outputs as they flesh out a film’s narrative arc in the early stages of storyboarding. Generative AI might also expand the “previs” process of transforming flat images of material environments and character interaction into 3D approximations of scenes.
Generative image software also offers artists new ways to connect with audiences. For example, German technologist Fabian Stelzer engaged with Midjourney, Stable Diffusion, and DALL-E 2 to create a core set of images for a sci-fi fantasy about the discovery of a malicious salt substance in outer space. He also used GPT-3 to assist with the script and voice-generators Murf and Synthesia for the narration. Stelzer then published short sequences of this film, SALT, on Twitter (what he calls “story seeds”), along with a call for followers to vote on further developments in the unfolding narrative. The result is a crowdsourced essay film that takes some loose formal and thematic inspiration from Chris Marker’s landmark experimental essay film, La Jetée (1962).
Generative AI also offers a way to blend graphics and performance, breathing life into genres such as the music video. Belfast-based computational artist Glenn Marshall won the Jury Award at the Cannes Short Film Festival for his AI-driven film, The Crow (2022). Marshall started with Duncan McDowall’s captivating video of Dorotea Saykaly dancing in a black shawl in a decaying industrial building. He then fed frames of the video to OpenAI’s CLIP, a neural network model. The result is a work of clever style transfer, where every live-action frame is given a painterly treatment that transforms the dancer into a crow-like figure, encouraging viewers to reflect on the relationship between human and nonhuman animals, technological process and aesthetic experience.
Part of what is exciting about these projects is their strangeness and messiness, the ways they flaunt their own fabrication. These projects point to the productive frictions of mixed-media and cross-platform practices.
Much of the utopian rhetoric around generative AI has to do what it means for the individual maker. Never before has an amateur or seasoned professional been able to build such an elaborate project on such a small budget in such a short amount of time. This line of thinking resonates with the long history of mythmaking within Silicon Valley and Hollywood, two places that love and love to loathe their geniuses.
The more promising dimensions of this software and the debate surrounding its use, however, have less to do with visions of reinvigorated auteurism than the fact that it requires a constellation of skilled labor, not only to make it, but also to deploy it in innovative fashion. We might see a broader reframing of authorship as VFX supervisors—not to mention the computer scientists, concept artists, engineers, and animators they work with—become increasingly responsible for the movements and expressions of the characters on screen, as well as the look and feel of the world they inhabit. It remains to be seen whether this is an inherent feature of this new media or a first stage, much like the early stages of cinema itself.