LTX Video In ComfyUI - The Fastest AI Video Model Run Locally - Tutorial Guide

Hello everyone, AI just never, never sleeps. And again, we have another new AI video model that's supported on the first day of launch, open source in ComfyUI, fully supported natively in ComfyUI. You are able to use the custom nodes or the ComfyUI default nodes to run this AI video model. Now, this is called the LTX video. Sounds familiar?

If you're a longtime subscriber, you may remember that I did a review of the LTX Studio video generator seven months ago. In that review, the AI models still used UNet architecture, which is not a diffusion transformer architecture. We witnessed a lot of morphing and short scenes, typically lasting one to two seconds. There were many close-up shots of characters, and even when the camera was panning, the video diffusion styles remained quite stable. I gave a rather unfavorable review at that time, as I felt that while the concept was appealing, it was not suitable for making AI movies.

Using one short text prompt, let's try to automate the entire process. For those involved in creating AI movies or documentary videos, you know what I'm talking about. Today, we will give it another shot with the new AI video models. Currently, the LTX video models are open source, and we can utilize them in ComfyUI, as previously mentioned.

The open-source downloaded weights of the models are available, and we can run them. In this video, we're going to check out some of the generated results using the LTX video and see how that looks. Meanwhile, I’m showcasing some of the videos I generated with this AI model over the last two days while experimenting with it. I can confidently say that this model exhibits pretty high performance. I’m amazed that this diffusion transformer model can run on a local consumer PC.

For me, I'm shocked that this diffusion transformer model is able to run on a local consumer PC. I'm using an NVIDIA 4090. I assume that a lot of average graphics cards can run this model pretty well and smoothly. In the announcement on X.com from Lightrix, the company stated that this AI model is capable of generating 5 seconds of video at 24 frames per second on the NVIDIA 4090, and that it runs very smoothly. I also saw one of their messages indicating that they can generate 4 seconds of video very smoothly using the NVIDIA H100.

So, imagine you have an H100, and you only need four seconds to generate AI videos of this quality. Sometimes, if you have a higher speed or better configurations of a GPU, you will even achieve better output compared to lower-end GPUs. Even lower-end GPUs can produce some results, but generating results doesn't always mean they are of good quality. We need to keep that in mind.

And here, as you can see, there are some text prompts that allow you to experiment. They may have trained some good results with specific video scenes. I see a lot of styles like that, and I have tested various results where a witch is cooking some magic soup. By using different seed numbers, I'm able to generate different styles of that scene. As you can see right here, my output duration is not five seconds; it's eight seconds.

So, there are some tricks that I used to extend the videos. As you guys have seen in my previous videos using Cloud Video Extend, I applied the same concept for the LTX video model. This is one of the video scenes that I posted on X, which received a lot of attention from the team. Hopefully, during this video, we can generate something similar using some text prompts, which will be displayed here, and see how that turns out. Here, you can see some text prompts that you can experiment with. Be sure to visit the GitHub project page, where you will find links to the LTX video, the website, model download weights, and some demos.

There are also some text prompts that you can experiment with. For example, one prompt describes a woman walking away from a white jeep parked on a city street at night. This was generated with the model I used for this result. In fact, this is the actual result I obtained, and I extended it to 10 seconds, showing the woman walking to the front door. There’s some movement, like her picking up the key and opening the door.

This type of movie scene can also be created, along with some landscape views, such as a camera panning over a snow-covered mountain. We will try some text prompts here. So, first of all, how do you use this AI video model? We are going to check out the Comfy UI page right here.

As you can see, the highlight of the LTX video in Comfy UI is its real-time generation speed. This means that this AI model is very lightweight, even though it is based on a diffusion transformer architecture. Despite being a 2 billion parameter size model, it runs very quickly on a consumer PC as well. I have tested the generation time, which is approximately 20 seconds to produce a few seconds of video. Therefore, this is a very consumer PC-friendly AI model that utilizes a diffusion transformer architecture. Additionally, the advantage is that these AI models are trained for 24 frames per second, which is the standard for making movies and high-quality videos.

Based on their highlighted key points, I would have some doubts about this. You know, sometimes we have to cherry-pick specific AI video scenes, even when using other very good AI models. However, it is natively supported in ComfyUI, meaning you don't have to download any custom nodes from a GitHub project or any external source. You can simply use this workflow for image-to-video conversion by updating your ComfyUI to the latest version.

Now, the last time I talked about the Flux tools and Redux, a lot of people asked some nonsensical questions, like why the tensor is generating errors, etc. Well, that’s because you have not updated to ComfyUI. You probably saw a new video about Flux, got hyped up, and rushed to try it without properly updating. This is what I refer to as a "zombie brain" moment. So, remember to follow these steps before asking any questions about nonsensical issues, such as index or tensor errors. Those issues are all related to the version of your ComfyUI installation. Please make sure to update before seeking help.

Update your ComfyUI, download the models, and place them in the models checkpoint folder. Then, use the T5 XXL CLIP models; here is the FP16 version. Put that in the models CLIP folder. Once you've completed those steps, you can grab the examples from the link I will provide in the description below. If you cannot find the official ComfyUI site, links will be included underneath this description. Currently, this AI model in ComfyUI supports two modes: text-to-video and image-to-video, which are very common for many AI video models today.

The other method is downloading the LTX videos and custom nodes officially launched by the LTX team themselves. However, I don't think I will cover this method because the latest versions of ComfyUI already include native nodes that support these AI models, similar to Flux or Stable Diffusion for running in ComfyUI. Therefore, I believe it is unnecessary to address the add-on custom nodes. By focusing on this basic process of converting text to videos and images to videos, we can utilize the ComfyUI workflow downloaded directly from the official ComfyUI website.

First of all, we need to load the checkpoint. As you all have experience with Stable Diffusion, many AI models that generate images, such as SDXL and SD 1.5, utilize checkpoint models. So, please place the saved tensors in the checkpoint model. This will be accessible through the provided link.

You can click here to be redirected to Hugging Face, where you can download the LTX Videos 2B file, which has a size of 9.37 gigabytes. Download this file and save it in your Comfy UI checkpoint folder. After doing this, you will have the selected option in your drop-down menu. Then, go to "Load Clip." Unlike Flux, where you might need a dual clip, you only need a single clip for this. Here, we have the T5FP16, which is the one you will use. In the second drop-down menu, we will select the LTXV.

That means we are using the LTX videos model, not Mochi, SD3, or Stable Diffusion—none of that. We are using the LTXV; that is the one, so only one CLIP loader is necessary. We don't need CLIP L or CLIP G or whatever that is. The next check is the CLIP text in code, which is very typical, similar to what we used to do in text-to-image.

Very simple. In ComfyUI, we don't need to explain too much about that. In this case, the sampler is different. We are using a custom sampler where we have the sampler sigma and latent image to handle. Because this is an AI video model, we have to manage some segments during the sampler processing.

So, that is the difference between using the K-sampler in ComfyUI. Additionally, there is a distinction between the condition, the text prompt, and the sampler condition. We have to add this LTXV conditioning for converting the frame rate and passing through the positive and negative condition data to the sampler for processing. Then, the MTL TXV latent videos will set the width and height, and the length will represent the number of frames that are going to be generated in the output videos. In this case, it is 97. You can imagine that if you want to generate around 5 seconds of video, you will need about 121 frames to create a 5-second length and other related aspects. Essentially, this process will be the same as what we used to do in the sampler.

So, regarding the case sampling selection, we are going to select the method of sampling. We don't need to configure anything, so don't touch that. The LTX scheduler is going to pass our empty latent data to the scheduler, and the scheduler is going to convert some segment data and pass that into the sampler's custom segment, along with the latent image data as well. We will have the generated result in the output latent data for the VAE decoder to perform the final decoding. We will get the generated output in the final video combined using the save animate nodes, which is going to work as well. I would prefer to use the video combined option because it offers more flexibility, and I don't need to save the input or output videos. You can also change the format to MP4, which will save it in MP4 format, and set the frame rate to 24. Then, you are good to go for this one. So first, let's try this without any modifications.

We don't need to make any modifications to see how quickly we can generate a five-second video. Usually, you might inquire about numerous AI videos created by our diffusion transformer architecture, which typically require a significant amount of GPU VRAM power. However, this particular process is very lightweight. I suppose they are utilizing something akin to turbo styles of generation for each image frame.

So that's why their processing time, as you can see right now while I'm talking, is already in the VAE decode, and it's very fast. However, one thing you need to be aware of is that you have to type fairly long text prompts for this AI model to generate good results. Without a very detailed description in your text prompt, this AI video may not produce satisfactory outcomes. So, provide as much information as you can, and this AI model will give you impressive performance, with a resolution higher than Cloud Video X. Okay, so we've got the result here. Yes, the girl is talking.

This is the text prompt coming from one of their testing text prompts here, and it looks pretty nice. I have not seen any morphing or deformation of any part of the woman's character. Now, let's check out the speed of this. This is the most important part of this AI model.

As you can see, I'm running 30 steps in the sampling, and I spend only 16 seconds on the generation of this AI model. The total time for running the prompt—starting from when we first load the AI model to finishing and displaying the output video—is 30 seconds for the whole process, including the initial loading of the model. That is with an NVIDIA 4090, as you can see right here. Now, let's try another one. For example, I like this mountain view. Let's see if we can create some output like the examples here.

So, copy and paste it here, and we can see the text prompt and run this again. Do you see the sampler's green bar? I have not fast-forwarded this video; it's just running at 1x speed. And we got the output result. Look at that! It's actually pretty nice.

The generated result, although it is not in high resolution, is the output of this AI model with a resolution of 768 pixels in width and 512 pixels in height. This is the standard size for this AI video model. However, if you would like to upscale these videos, you can certainly do that to enhance the quality further, sharpening the colors and applying extreme upscaling. The result can effectively use text-to-video capabilities, and there are some more complex prompts included, such as those featuring people talking, as seen in this example.

So, two women are talking, or this prison guard is unlocking something. This is a pretty complex prompt as well. As you can see, it's very long. The text prompts used in the LTX video model will be significantly longer than what we’re used to. Therefore, I suggest that if you’re the type of person who doesn’t like to type long text prompts, as I do, you use a prompt enhancer or large language models to help generate longer text prompts.

Please provide more details and be more descriptive in your text prompt. Here we go! We've received another result. Although we won't have the same video scenes due to the different seed numbers, the overall outcome is quite nice.

We have 24 frames per second speed, which looks good, and the motions are smooth and fast enough for the normal movies or videos that we can produce. I would say this AI model can be fine-tuned, or if you have a more detailed enhanced workflow in ComfyUI, you can create it to generate a better quality result. I have found one important aspect regarding the step numbers. If I increase it to 50, it generates something of higher quality, since we will only be using 16 or 17 seconds for each generation.

So even if I set the step numbers a little bit higher and some longer lengths, for example, if I set it to 200 frames for each generation, I don't think that really matters for the time—at least for me and my computer system. You know, with the normal settings, I use about 17 seconds or 16 seconds. So if I adjust it to 50 seconds, six steps, and increase the frame numbers, I'm still able to accept that longer time for the generation of the video. Let's check out the video result once it is done. And there you go.

We received the same text prompt from the prison guard, this demo text prompt, but then we switched to 50 seconds and 201 frames, which consumed close to one minute. That amounts to 55 seconds in this generation. Now, let's take a look at the result. Compared to the last one, this is, of course, more detailed—indeed, even better.

We have a sharper face. It's not going to be a pixelated, larger face where we have more detail present in the background environment. You can also see that the scenes show the guy turning to the right side and then going into the prison, where it will be more detailed and more realistic in style, using this text prompt setting with higher step numbers for the sampling. I would recommend setting these steps to 50 for the LTE X videos. Of course, you don't always have to set a high video frame length here, because by the standard of this AI model, it generates 5 seconds.

So, let it be just 121 frames. That is a normal generation length for these AI models. Let's try one more text prompt, which is going to be another human character-like video scene. We will input this prompt here, click generate, and check out the result. This is very good for a locally running AI model or smaller parameter size models, such as those with 2 billion parameters. And this is done within 29 seconds.

As you can see right here, I'm using 29 seconds because I have set a very high number of sampling steps, which allows it to generate a high-quality, human-like character. In the talking scenes like this, you can see the head shaking. There are one or two moments where the movement of the head looks very realistic, and all the elements on the head are quite lifelike. The jacket here is very coherent, and I haven't noticed any deformations in the human character either.

Although the resolution is not high, it is still in 720p. However, we can upscale it if we want to. So, I have also tested the image-to-video workflow, which has a very similar structure to the text-to-video workflow in terms of node connections. The setup is almost the same, except we start with loading our image and incorporate some new nodes from this ComfyUI update, specifically the LTX Image to Video nodes. As you can see, these nodes come by default from ComfyUI itself and are not external third-party custom node installations.

We are going to try some image-to-video functionality here as well. However, I can assume that this AI model may not perform very well in image-to-video generation. First of all, let's try out this landscape view. As you can see, a text prompt is already preloaded in this workflow, but we can explore other options as well. I will be using a third-party tool to create captions from this image and then using those captions for the Florence 2 model.

We are going to use that for the text prompt here. However, I want to keep this preset text prompt for the quality rise settings. Therefore, we will combine another text prompt. I would like to use combined text, specifically the CR combined prompts, which have a few text areas available.

I will be connecting the captions for this one to the Florence 2 running on top of here. We will send the captions to Part 1 and then send the combined prompts. We need to set the prompts. All these good things, text prompt presets for Part 2, and then we will switch the text encode nodes for the text to be input parameters. We will send that text prompt here. After we combine all this text string and pass the string to this text prompt as the input, we will generate the result.

And then, I would like to set the sampling steps higher, like 50. That should generate something of better quality. So, let's check it out. One more thing: I don't like the animate nodes that come by default with Comfy UI, so let's use video combining here. Set the frame rate to 24, which is paired up with this AI model. Then we can try it again, setting it to five seconds, and see how this image turns out. Okay, we have the result here.

For the first result, I assigned fairly easy tasks to this AI model, starting with the simpler ones before moving on to the more difficult tasks later. The initial output looks quite good. The water from the river flows in one direction, and it appears as if the camera is shaking, possibly due to someone holding it. We can see that the model is functioning well, as it can process all this text prompt data and send it to the sampling stage to generate our results. Now, let's try another image and see if there are any issues or if the AI model struggles with anything.

Let's revisit the last time we used Cloud Video X. We had the robot sitting on a cyber flying device. We can try this one. Okay, so we got the results from here. We can see the animations of this robot flying object are very nice. Actually, I like that the clouds in the background are animated. When we tried this image in Cloud Video X5B, it was unable to make the clouds move.

But of course, in this AI video model, it's also able to animate the objects. The burning fire behind the missile device adds to the scene, and the robot can tilt slightly while sitting on this flying object. This looks quite nice. As we have explored the AI videos, we've noted the limitations of the five-second duration, and we want to extend that. So, I applied the previous video extension method and incorporated those ideas into the LTX video workflow. First, let's turn on all the groups to see how that runs. I set up the image-to-video workflow groups here, which are very similar to the demo workflow groups you just saw. However, I've made some modifications and changes in this area.

In this section, I always like to use the "Load Image" or "Load Image from Path" options. This way, I can choose whether to load an image directly or to disable it and use a path instead. After making my selection, I set the sampling step, which is always set to 50. Then, I use the text prompt area, where we can focus on a very specific input for the text prompt, and this will be passed to the entire generation process in this AI video workflow. Another section is for text-to-video. This is a straightforward process—simply type the text prompt at the top, and it will pass that string to the designated area, starting the video processing, as I just demonstrated in the text-to-video workflow. And, of course, the exciting part is the video extension feature.

This is what I like to use to extend another five seconds. In my previous demo, you saw a person walking in a car park from a white truck, then walking to the door and picking up the key, trying to open the door. I did this using the video extension feature. As a result, I generated a total length of ten seconds in this video scene. Lastly, we combine those two videos, with the first one running from the text.

From the image to video, you can choose either option in the first choice here: an image video result or a text video result. Select the one you are using. Additionally, the video extension will be joined together, stitched back into one long video. As I do in other workflows, I always use the group bypasser to enable and disable the groups that I need to run.

So, if I'm using the image-to-video group, then I'll use the group bypasser to enable and disable the groups that I need to run. If I'm using the image-to-video group, then I'll use image to video. And, of course, the first thing is to turn off the video extend; we don't need that at the moment. We generate the videos here first until we see a good result in the combined video output, and then we'll bring that result to video extend. So, let's say I have scenes of a woman turning her head. It looks like it's in a haunted house, where a candle lights up and suddenly the light in the room turns off.

So I put the text prompt here, and let's try to generate something in the image-to-video group. I have turned off the other groups as a feature to focus solely on the image-to-video group at this moment. Okay, so we got the result here. The woman turns around suddenly and then continues walking into the dark room. We received this result, but it's not very impressive in terms of the turning angle of her face; there is some morphing in the half-second moment. However, it is still acceptable overall regarding the appearance of the clothing. Additionally, the other objects in the background are well-formed and not defective.

Oh, I would say this is an okay video, but it's not really high-quality. If you don't like these videos or something, you can, of course, change the seed numbers and run this generation again. Okay, so let's say I have videos that I want to produce for a longer length, like this one. I have five seconds of video. So what I can do is go to the video extension, turn on those features, and then enable the final result as well.

Then go to the video extension, turn on those features, and enable the final result as well. After that, proceed to the final result. We will need to choose the first video to stitch together, which is the image-to-video result. That is correct.

And then we got the video extension to stitch the extended length. So, we are combining these two into one video length for the final result. The second thing we need to check here is to ensure that we get the video from the image-to-video result, which is correct. If you change to text-to-video, then you will need to change the options here.

Then the remaining features and settings will be the same as those shown in the image to video above. Most likely, they are identical, so we do not need to change too many settings here. In fact, the LTX video models are quite simple in terms of some custom nodes, making them very user-friendly. Once again, please confirm that everything is functioning correctly.

And remember to set your seed number to fixed in your image-to-video groups. If you set your seed number to random, the next time you use it, even if you like the result, activating the video extend feature will change your random seed numbers. So, be aware of that. Make this fixed, and also set the text-to-video to fixed.

The only random seed number will be the video extend groups, which is located here. So, click one more time, and we will see how the video extend result looks. We'll have the full length by combining these two videos together into a 10-second video. Okay, we have the complete generated result here. Now, let's check out the videos, starting with the video extend.

It is from the last image frames of the image-to-video group, where we see the woman walking near the windows. After that, we observed the woman in the video stepping back two steps, feeling scared, and looking around as if she’s shocked by something she saw. Additionally, there are some candles generated by the AI in the video, which are not part of the initialized image frames we loaded. Therefore, there are more scenes in this video generated by the AI itself.

It's also able to maintain the same styles as the image I provided as the initial reference. When you look at the first image at the top, you can see the blue curtain from my image, and it begins to transform into another angle. Usually, some AI videos that are not well-trained tend to morph or change colors unpredictably; however, this one maintains a coherent style. After I extend the video from the last frame of the woman walking near the window, it is able to produce a video in the same style.

The window curtain, which is blue, along with the wall, can produce the same styles as the image I captured from the last frames of the video. That is pretty impressive, I would say, for a local AI model that can run on your local machine. If your PC has at least 8 GB of VRAM or higher, you may be able to use this AI model on your system, especially if you are using one of those older PCs from a few years ago. However, I'm not sure what the minimum requirements are; they may or may not announce them. From their cross-post, it appears that they are using an NVIDIA H100, which consumes just 4 seconds to generate a video.

So that is pretty nice; that is what we should have. As you know, for video generation, we need something that runs fast. It's not only that the slow motion consumes a lot of our VRAM, but we only generate like 1 or 2 seconds of video. Those are ridiculously small outputs. So, I think this is a good video model.

I will keep this in my ComfyUI system. I will not toss this one. This is the LTX video, and they have lots of sample demos here. So, I will see you guys in the next video. Have a nice day!

See you.