How to automate AI video creation?

5 min readMay 29, 2023

Hello!

You’ve probably already seen a bunch of AI-generated images and videos. Have you ever wondered how they’re created? Today, I will share with you the entire process of how you can create a similar video using just two free programs and your laptop.

In this article, you will learn:

What the results look like
How to set up and run the programs
How to control the behavior of the AI
How Instagram users react to AI-generated content

Let’s also conclude with a full video.

Results of Video Creation

We have two different results.

Option 1 involves feeding video frames to a stable diffusion with a static prompt, essentially acting as an advanced filter for the video.
Option 2 involves feeding the freshly generated AI image as input for a new one and modifying the prompt on the fly. As you can see, the AI can go quite wild, so the parameters should be carefully chosen.

Setup and run

Before we dive into settings and code, let’s look at what to install on our local machine. I use a Mac M1, but this should work for Windows as well.

Step 1. Install Stable diffusion software:

https://github.com/AUTOMATIC1111/stable-diffusion-webui

Follow the Readme instructions and then run it locally:

./webui.sh --share --skip-torch-cuda-test --no-half

Now you have your local version of the Mid journey. Play with the UI, and whenever you’re ready, continue with Step 2.

Step 2. Clone and run the custom script:
I built a simple NodeJS program (https://github.com/skobak/video-to-ai) that takes a video as input, along with parameters, and produces a sequence of frames modified with AI.

# get copy of a program
git clone https://github.com/skobak/video-to-ai
# open the folder
cd video-to-ai
# we need to have node >17
nvm install 17
nvm use 17
# install deps
npm i 
# run programm with default settings
node main.js

Now, you can enjoy the process of creating your first AI video with automation!

Settings and Configurations

As always, the result comes with experimentation. For Stable Diffusion, two main parameters significantly affect the results:

CFG Scale: Affects how wild the AI will go with a prompt
Denoising strength: Affects how far it will deviate from the original image

You can play with these results by providing flags: -c and -d.

#example
node main.js -c=14.4 -d=0.5

Let’s look at how these properties affect the end result:

Original(c=0,d=0), Slightly modified(c=7.5,d=0.3), Wild(c=25.4,d=0.6)

Feel free to play with script parameters:

-v <video path>
-p <prompt>
-i <prompt input file path> !NB, this flag will run Scenario 2
-f <number of frame>
-c <CFG Scale, creativity scale 1-30>
-d <Denoising strength, 0-1>
-s <seed>

Example:

# This command will create an AI version of each frame of video.mp4
node main.js -p="Make it look like a painting in modern style" -v='./video.mp4'-c=14.4 -d=0.5

Or, go to ChatGPT and use the following prompt:

Make a 25 prompts, one per line separated by comma in CSV format 
that will generate the different style of painting of a cat in different artistic styles

Copy the result into prompts.txt file in a root of script folder and then run:

# This script will take last frame of a video and create 25 additional one with
# Maximum of creativity
node main.js -i="./prompts.txt" -v='./video.mp4' -c=29 -d=0.5

Here we go, enjoy the result!

How Instagram users react to AI-generated content

What about the product side of our experiment? We’ve been developing our cat’s Instagram account and uploaded normal and AI-generated videos as reels to find out how they perform:

I haven’t found a direct correlation — simple reels can go viral without any AI. However, AI can serve as a useful tool in your toolbox.

Conclude/Full Video

https://www.youtube.com/watch?v=2EZni7CcXeM

Today, we acquired knowledge on automating AI video creation, identified the tools necessary for the process, and learned how to construct a custom script. My main insight is that in the present era, we can swiftly develop extraordinary tools by merging AI with automation and scripting.

Next time, we will test and refactor our code using Co-Pilot and Chat GPT. Our goal is to discover methods of making our solution not only easier to maintain, but also thoroughly tested.

I hope you found this intriguing. Please feel free to provide feedback! I aim to delve deeper into AI tools, both here and on my YouTube channel.

Don’t forget to subscribe and share!

Cat account: https://www.instagram.com/alice_tigers_paw/reels/
Source code: https://github.com/skobak/video-to-ai
My youtube: https://www.youtube.com/@alex_se

Best,
Alex and ChatGPT 4 assistance 🤖