Evaluating Six AI Video Generators for Animating Original Characters: A Comic Artist and Animator's Experience

hilary price

Summary:

Hilary Price, a comic strip creator, and Sean Campbell, a 2D animation major, conducted an experiment to animate Hilary's original characters (an inchworm and an ant) using six different AI video generators.

• The Challenge: To animate custom characters swinging around a tree, going beyond simple text-to-video prompts, requiring dynamic, multi-dimensional motion.
• AI Models Tested: Google's Veo, Luma, Wan, ChatGPT's Sora, RunwayML, and Midjourney.
• Common Issues: Most AI models struggled with maintaining character consistency, unintended "talking" mouths (even when explicitly instructed not to), and precise motion, often requiring multiple iterations or revisions.
• Specific Model Performance:
  • Veo produced mixed results, showing the "talking mouth" problem.
  • Luma allowed revisions but eventually led to distorted characters.
  • Wan, despite its AI prompt revision, took hours and had character inconsistencies.
  • Sora and RunwayML were deemed more suitable for image generation than robust image-to-video animation for custom characters.
  • Midjourney provided the best starting point, even understanding metaphors like "Spider-Man swinging."
• Human Intervention: Midjourney's output still required significant manual cleanup and frame-by-frame revision in Adobe Animate to achieve a polished result, including correcting distorted faces and removing unwanted mouth movements.
• Reflections: The project highlighted AI's potential to reduce animation workload but also its current limitations and the ethical implications of "learned" animation styles. Collaboration between human artists and AI proved most effective.

Introduction to the Project [0:00]

• Project Participants [0:07]
  • Hilary Price: Creator of the daily syndicated comic strip "Rhymes With Orange," currently working on an illustrated middle-grade novel.
  • Sean Campbell: A 2D animation major at Rochester Institute of Technology, who joined Hilary as a summer intern for this research.
• The Animation Challenge [0:28]
  • The goal was to animate original characters, an inchworm and an ant from Hilary's novel, swinging around a tree using AI video generators.
  • This project aimed to explore animating pre-created characters, moving beyond typical word-prompt generated AI videos.

AI Video Tools Explored [0:50]

• The team selected six generative video AI models most relevant to their research: Google's Veo, Luma, Wan, ChatGPT's Sora, RunwayML, and Midjourney.

1. Google's Veo [1:12]

• Initial Test with Simple Prompt [1:12]
  • Prompt: "An inchworm swinging from its silk and an ant on its back holding on like Tarzan. The style of the characters is cartoony like you'd see in a comic book."
  • Result: Veo provided two videos, with one being a slightly better result, but still struggled with accurate character representation and motion.

    

• Ambitious Dynamic Animation Test [1:35]
  • Sean redrew the characters from multiple angles to create a style guide, helping him understand their functionality and identify AI mistakes later on.

    

  • He also drew specific beginning and ending frames required by AI models for dynamic animation.

    

  • Prompt: "A caterpillar swinging around a tree using its silk as a rope. An ant is holding onto its back like Tarzan. As the inchworm swings around the tree, it lands on a bundle of leaves sticking out and poses triumphantly."
  • Veo's First Result [2:22]: The AI frequently made the characters' mouths open and close, despite no instruction for them to talk, a common issue observed across several models.
  • Veo's Second Result [2:41]: Similar to the first, demonstrating the persistent "talking mouth" problem.

2. Luma [2:54]

• Luma was unique for allowing users to revise and work off previous video generations, unlike others where changes meant starting from scratch.
• Initial Test with Waving Animation [3:05]
  • Prompt: "A tall, green inchworm is standing upright on a pair of leaves. Holding onto its back is a small ant whose head is poking out and is visible to the viewer. In a short animation, the inchworm and the ant wave at the camera while smiling."

    

• Revision 1 [3:37]
  • Revised Prompt: Instructions to remove eyelashes, eliminate talking, smooth the frame rate, make the ant's face more consistent and less expressive, and depict a slow, controlled wave.
  • Result: Luma "stood firm," largely ignoring the revision requests, especially regarding the eyelashes and talking.

    

• Revision 2 [3:59]
  • Using the same revised prompt.
  • Result: The animation became "weird," with the characters appearing distorted and merged.

    

3. Wan [4:28]

• Wan had the capability to revise user prompts using its own AI.
• Basic Swinging Prompt [4:30]
  • Original Prompt: "An inchworm and an ant are swinging from a rope of silk attached to the inchworm's rear. Backgrounds transition across the screen from right to left as they swing forward. First a forest, then a lake, and lastly a swamplands area."
  • Wan's AI Revised Prompt: This automatically generated a much longer, more descriptive prompt, including details like "surprised expression," "petite purple ant," "mouths stay shut," and specific background transitions.

    

  • Result: Wan took hours to generate a video, which ultimately showed inconsistent character appearance and animation, falling short of expectations.

    

4. ChatGPT's Sora and 5. RunwayML [5:55]

• Both models were tested using the basic swinging prompt.
• Evaluation: While these AI models are highly capable of generating high-quality images from scraped internet content, their current image-to-video tools were not considered "up to speed" for animating custom characters with the desired consistency and motion.

  

6. Midjourney [6:18]

• Midjourney generated four results per prompt.
• Initial Swinging Prompt [6:20]
  • Prompt: "A frame by frame animation of an inchworm swinging around a tree using silk as a rope. An ant is holding onto its back and is visible from behind. They swing around the trunk of the tree and land on a small branch of leaves, where they pose triumphantly while standing up straight."
  • Results (Versions 1-4): The generated animations often depicted the inchworm "surfing" on its silk rather than swinging naturally, but overall, Midjourney produced better outputs than the other models.

    

• Spider-Man Metaphor Test [7:16]
  • The team wondered if AI would understand a metaphor for the swinging motion.
  • Prompt: Modified to include "the inchworm holds onto the rope like Spider-Man," along with instructions for camera movement and a specific command: "The characters do not open their mouths or talk at all."
  • Result: Midjourney understood the "Spider-Man" metaphor for the swinging action, improving the motion. However, the persistent problem of characters opening their mouths and "talking" still occurred.

    

Refinement with Adobe Animate [7:58]

• Sean took the best output from Midjourney into Adobe Animate for post-processing.
• Manual Revisions: He meticulously revised the rough spots frame by frame, which involved erasing the extraneous silk rope, removing unwanted mouth movements, and correcting distorted faces that the AI produced.

  

• Final Result: The combination of AI generation and human artistic refinement resulted in a much cleaner, more accurate, and polished animation.

Reflections and Conclusion [8:24]

• Sean's Perspective: Initially skeptical of AI, he now acknowledges its potential to alleviate workload in the animation industry. He remains on the side of human artists but anticipates future AI versions that are less intrusive on human creativity.
• Hilary's Perspective: She found it delightful that AI could bring her characters to life. However, she raised concerns that the "learned" swinging motion by AI might be derived from existing human-created animations, questioning whether AI will genuinely assist or eventually replace young animators. She emphasized that the collaborative learning experience with Sean was a rich human interaction that AI cannot replicate.