May the 4th Be with You: The AI Behind Movie Magic

Whether you adore the Cantina Band or prefer to jam out to The Imperial March, we’d like to say… May the 4^th be with you. May the 4^th(Star Wars Day), is an informal holiday that honors the Star Wars legacy and its undeniable impact on modern filmmaking. Star Wars is credited with a host of innovations in cinema including the creation of expanse universes, movie franchising and merchandising, and the biggest impact (that we are eternally grateful for) of visual and special effects. Up until A New Hope was released in 1977, movies were heavily focused on dialogue versus special effects. George Lucas’ creation of Industrial Light & Magic (ILM) allowed for these effects to play a bigger role in film and for AI to eventually take center stage creating movie moments that we’ll never forget. Advances in Computer Generated Imagery (CGI) today allow content, video, and images to be entirely computer generated. The first instance of AI in A New Hope was through the techs at ILM building a computer-controlled camera system. Coupling that with custom built processors, they were able to replicate camera movements, to which George Lucas said they granted “cinematic freedom” and had “a very powerful impact on the storytelling.” Forty years later, technology has evolved such that machine learning was used to create one of the most infamous Marvel villains of the 21st century, Thanos. To accomplish this effect, tracking dots were placed on Josh Brolin’s face to capture his expressions. That data was later used to train the AI model to replicate them. That’s just one example of a stunning character created using AI. Other iconic characters include Gollum, King Kong, and Yoda. With AI, production companies can program characters with whatever characteristics and expressions will appeal most to a certain target audience, increasing the chance of engagement.

AI is Used Throughout the Movie Making Process

AI is leveraged throughout the whole production of modern cinema, from speeding up the process of pre-production to putting the finishing touches on a film. You may not immediately think of it, but AI helps with all of these film-making processes:

Scriptwriting
Visual Effects (CGI)
Scheduling / pre-production
Subtitles
Editing

It’s important to note that, while AI can do all the above, humans are essential in monitoring the process from start to finish. A human element is still required to add that magic that makes movies entertaining and relatable. Scriptwriting, for example, is largely credited towards writers coming up with the idea and working with a team to create the next blockbuster. There’s a notable case from 2016, however, which was the first time AI wrote an entire script on its own. What resulted was a 10-minute short called Sunspring using a model trained from scripts produced in the 80s and 90s. Despite a few shortcomings to using the AI (the model couldn’t understand names and gave stage directions that were impossible), the film placed in the top 10 at the Sci-Fi London film festival. AI is also used by production coordinators to review actors’ schedules, analyze the data, and generate a custom schedule so actors with small parts can film their scenes in a short time frame. This saves production companies time and money by simplifying the travel and production schedule. You’ve likely seen auto-generated captions on social media videos–it’s a growing trend that allows for translations and for the hearing impaired to engage with content. Natural language processing (NLP) and similar technology is being used to create subtitles for movies, allowing the film to be accessible to viewers worldwide. To date, Star Wars has been translated in more than 50 languages, and the list continues to grow as technology improves and the franchise remains a favorite. Humans are still needed to ensure the subtitles are generated correctly. Experts say that watching movies in another language is a great way to learn, so next time you want to watch Star Wars stream it in another language and you’ll pick up some new words. While the capabilities of AI on set and in the editing room are truly impressive, CGI is where this technology really shines. When paired with a green screen, the background of a scene can be virtually anything! The same holds true for placing full CGI characters in films. Even for fully rendered characters, there is a human donning a green suit allowing the CGI character to be placed in the correct spot. Production studios also use CGI to bring actors to life that have passed away before they finished filming their respective scenes. In fact, this was done with two beloved Star Wars characters, Carrie Fisher (Princess Leia) and Peter Cushing (Grand Moff Tarkin). The first instance was for Rouge One as that story is set at the same time as A New Hope but was filmed nearly 40 years later. CGI was used to make the actors look just like they did in A New Hope. The second instance was for Episode 9 – The Last Jedi. Carrie Fisher had passed away before finishing her scenes, and some CGI was used to be able to complete her story.

How Appen Makes Our Own Movie Magic

If you think this is fun, you’re right, and we’re so excited that we can be part of it. It’s common for successful films to create merchandise, including some of your favorite video games. We’ve had the pleasure to work on AR/VR worlds with well-known software companies and gaming systems. Our role is to train AI models to read the movements of a video game player and have that action happen in-game. We achieve this by teaching AI models to determine if there are any body parts in their image range and what motion they’re doing, similar to the process used in creating Thanos. As you can see, the world of AR/VR isn’t just limited to video games. Other AR and VR use cases:

Detection, recognition, and labelling for objects, audio, and text
Procedural content generation (similar to CGI in that we create characters, environments, or other objects to scale)
Virtual humans
Embodied interactions (mimic human movement)

Just like AI assists with subtitle generation for movies, Appen also works on translation, localization, and Natural Language Processing (NLP) . We use AI to aide in generating text from dialogue or translating text from one language to another. After the initial work is done, our team of highly skilled linguists ensure everything is correct. While we may not be making movies, we can certainly help you achieve the visual effects needed in yours.

Appen’s Thoughts on the Future of AI in Film

As technology improves, we believe AI will have an increased impact on film. Two areas we anticipate seeing significant improvement in is scriptwriting and subtitles. The AI that created Sunspring, has made two more movies since. In the future, we expect to see more AI models writing scripts (and ones that can create character names and proper stage directions). These scripts have also been limited to 10-minute shorts. The next feat may be writing a 30-minute script, and eventually a full-length feature. As AI models continue to be trained daily with more data on language translations and dialogue to text transcriptions, we anticipate increased accuracy. It won’t eliminate the need for humans to review the end results, but it will allow for a reduction in time spent. This can also lead to films generating subtitles in more languages, thus inviting a wider audience to enjoy each film that is produced. Without AI the modern-day film would be drastically different. We owe it to George Lucas and his team for being one of the leading pioneers to set the stage for the future of film and for creating an ever-expanding universe that has brought the world together. May the 4^th be with you this Star Wars Day, and remember, Han shot first.