From video synthesis to soundtrack editing, everything is done by AI: The trailer of the first AI sci-fi movie "Genesis" debuted shockingly

Source: Heart of the Machine

Author: Du Wei, Zewen

In the past two days, a sci-fi movie trailer "Trailer: Genesis" ("Genesis"), which is less than a minute long, has become popular on social media.

Jump to watch‌

Is it very sci-fi? What's more "sci-fi" is that from image and video synthesis, music to editing, everything is done by AI.

Producer Nicolas Neubert lists the corresponding AI tools, where Midjourney processes images, Runway processes videos, Pixabay processes music, and CapCut edits videos.

Midjourney is a familiar AI drawing artifact, and it has been updated to version 5.2. Runway is an AI-based video production tool, and Gen-2 is currently available for free trial. The CapCut editing tool is free for everyone, but you can also choose to edit in Adobe Premier or Final Cut Pro.

Twitter @iamneubert

It is understood that Neubert spent 7 hours, Midjourney generated 316 prompts, enlarged 128 images, Runway generated 310 videos, and a video with text. A total of 44 videos were used in the trailer.

Today, Neubert even wrote a long article, detailing the production process of "Genesis", including the specific process and how to use the above AI tools. Let's take a look at them one by one.

Regarding the idea of the movie, he said that the idea of his dystopian theme came from several movies he had watched, and he wrote a story based on them.

The first step in official production is to build the world and story.

For the trailer's Genesis storyline, Neubert wanted to build up the tension step by step. The following 3 phases are therefore defined:

1. Setting the scene
2. Introducing Threats
3. Climax in CTA

Specifically, Neubert worked on the first draft of the trailer’s copy, which included “Share It All, Live the Consequences, and Call Humanity to Action.”

Having thus defined the overall tone, he proceeded to generate scenes around these themes. Neubert scrolled through tons of human and sci-fi clips on themes of environments, military technology, and combat, and gleaned a story from them all.

Also to add some depth, he added shots of three kids with glowing amulets, hinting at a deeper storyline.

The second step is to generate consecutive images in Midjourney.

Pay special attention to the hint() here. Neubert refined the stabilization cues he'd gotten in previous posts to create a template so he could reuse it for every shot in the trailer. The template is as follows:

___________, Star Wars, styled as detailed crowd scenes, earthy naturalism, teal and yellow, frostpunk, interior scenes, cinestill 50d—ar 21:9—original style

For each scene, he would fill in the blanks with his desired scene, making sure that the other tokens maintained maximum continuity of theme, color and lighting.

In addition, the use of strong changes (Strong iations) feature makes it easier to create different scenes while retaining the previous color palette. A scene of a female warrior can be transformed into a scene of a normal citizen, a cyber hacker, or a fight without generating new prompts.

The third step is to generate the animated image in Runway.

Neubert found this step to be the easiest. On the setup, he always tries to activate the Upscaled mode. However, this mode often has problems with faces, so for portrait shots, he usually uses standard quality.

It is worth noting that instead of using a combination of text prompts and image prompts, he dragged and dropped an image and regenerated it until he got the result he was satisfied with.

The last step is post-editing in CapCut.

While Midjourney and Runway generate output, Neubert starts by placing key scenes that he knows will play a big role. As far as the trailer goes, he thinks the exterior shots will be the opening.

Then start planning the text. It is possible that there are no clips in the timeline when the text is positioned according to the music. In less than an hour, he compiled the content according to the timeline and fixed the location. This is great for generating images where you need an extra fixed point to account for which scenes are still missing.

The specific steps become very simple, generate clips → pull them into CapCut → place them on the timeline, and slowly piece together the story. He also color-matched 2 or 3 edit packages to make them look more like grand movie sets.

The only skill required to use CapCut is tempo-syncing the clips. When "BWAAA" comes up in the music, he's always trying to connect the action within the clip or line up the clips that follow. This makes the entire sequence feel more immersive.

In addition, Neubert considered how to incorporate a lot of motion into the clip. Here he uses two tricks to add movement.

First trick: Runway takes an image and calculates which parts should be animated based on the model. He reverse-engineered this idea, trying to output images in Midjourney that suggested motion. This means motion blur can be added to footage or still images of moving heads or people can be captured.

Second tip: When you analyze the Runway video, you will find that in the 4 second clip, the scene often changes greatly. So in the trailer scene, he only used the full 4 second cut twice. All other clips are 0.5-2 seconds long and speed up clips by a factor of 1.5-3. The reason for this is that, as a viewer, you only see a short clip and therefore perceive more motion in the scene, essentially fast-forwarding that part.

After some operations, what we finally presented to everyone is the shocking "Genesis" trailer at the beginning. The trailer also received rave reviews, with some saying it was the best runway generation video they'd seen so far.

In fact, after Runway Gen-2 was available for free trial, many netizens opened their minds and combined it with Midjourney to create boldly.

Midjourney+Runway: A magical combination of AI creation tools

Here are some other generation use cases to share with you.

Runway's grasp of the details of the character's movements is also relatively accurate. In the video of netizen Shencheng, it can be seen that the details of the eyes of the characters make the dynamic video more vivid, and it can also be said to add a bit of "acting skills".

source:

After the picture moves, the movements of the man and the horse in the night are very natural, and there is more room for imagination of the characters and even the follow-up actions.

source:

The combination of Midjourney and Runway looks invincible, and it can portray a proper sense of story in the key actions of the characters.

Twitter: @ai_insight1

There are also some variations that are richer and more creative in generating results.

Twitter @kkuldar

Twitter: @Akashi30eth

Some netizens also use a series of AI tools to generate video clips, but the results seem to be unsatisfactory.

Twitter user @nickfloats uses ChatGPT to write scripts, Midjourney to generate images, Runway to generate videos, Triniti to generate music, Eleven Labs AI to dub, and the clips are simply stitched together without other adjustments.

Script written by ChatGPT

Netizens recorded the whole process of their own production, and wanted to see how long it would take to use AI to make short commercials.

This netizen commented on the results of this short film as "The ad sucks". In this advertisement, the movements of the characters are stiff, the logical relationship of the picture is not strong, and it is difficult to highlight the theme to be expressed as a whole. The cooperation of AI seems to have no surprises, but more "scares".

Compared with "Genesis", this short commercial film is half as long, but it takes very little time. One of the biggest differences is in the details of human involvement. It lacks carefully considered themes, coherent adjustments, and attention to detail.

If only AI is allowed to participate, or only to produce works based on the generation of AI, it is obviously not possible to produce high-quality results. It is their application and adjustment by humans that seem to reveal the true value of these tools.