VideoAdAgent Bench v1

Headline — all arms as 0–1 scores

Why we built this benchmark

Until now, there has not been a public benchmark that grades AI video systems on the end-to-end task of producing a finished ad. Existing video benchmarks — VBench, Video Arena, EvalCrafter — measure per-frame fidelity on single-shot text-to-video output. None of them measure whether the output would function as an ad: hook strength, brand alignment, on-screen typography, hallucination cost, production polish. VideoAdAgent Bench fills that gap.

A stunning eight-to-twelve-second clip is not an ad. An ad has to hold a brand at the right angle, render the product name without garbling letters, build a narrative arc with a hook in the first three seconds, hit a fixed duration and aspect ratio, carry voiceover or music, and survive a production QA review without product hallucinations, anatomy glitches, or text artifacts that would force a re-roll. Today's text-to-video models do not attempt most of these problems. The agents that wrap them either do, badly, or skip them entirely. The point of this benchmark is to measure exactly that gap.

Mirroring real ad production

Each brief is structured the way our internal creative team writes one for production: product description, brand guidelines, narrative intent (problem-solution / before-after / demo / testimonial / lifestyle), platform constraints (duration, aspect ratio, required on-screen text strings), and the input assets the agent should ground itself in — logo, product reference image, persona reference. The agent receives the brief and returns a finished MP4. We grade the MP4 the way a creative director grades a campaign before approval: not against a checklist of features, but against whether the output is shippable.

The brief set spans every major ad-production pattern plus targeted stress tests against the hardest axes of multi-scene production: anchor storytelling, persona consistency, brand-asset reuse, structured CTA typography, and persuasion-arc compliance. Every brief uses fictional brands and AI-generated product mocks to avoid real-customer IP. The full set is pre-registered — published to a tagged commit before a single output was generated — so no system in the benchmark could have been tuned to the specific briefs.

Arm 1 — Agent vs Agent (Creatify vs HeyGen V3, Luma)

Strictly agent-vs-agent comparison. Both systems get the same brief and produce a finished ad; LLM judges (Opus 4.7 + GPT-5) see the brief and score 8 ad-rubric dimensions. Win-rate = % of pairs Creatify or HeyGen wins outright (consensus across both judges + both seat positions). Per-dimension scores = average score the judges gave each system across all pairs (un-swapped via position flag).

Arm 2 — Ad Quality (Creatify vs raw T2V baselines)

Multi-judge pairwise where judges see the brief and score 8 ad-rubric dimensions. Per-dimension scores below are per-system averages across every pair the system appeared in. Competing agents (HeyGen V3, Luma) are evaluated separately in Arm 1; this arm contains only Creatify and the raw text-to-video foundation-model baselines.

Two baseline variants per foundation model. Single-scene mirrors how Veo 3.1, Kling 3.0, and Seedance 2.0 are actually used today: the brief becomes a single prompt, the model returns one 8–12s clip. Multi-scene is a fairer variant we built for the baselines: an LLM decomposes the brief into per-scene prompts (the same decomposition step Creatify Agent does internally), each scene is generated separately, and the clips are concatenated.

Arm 3 — Video Production Quality (Creatify vs raw T2V baselines)

Arm 3 grades the finished video as a deliverable, separate from how well it answers the brief. Two complementary measurements: (1) a hallucination & visual-defect score from Claude Opus 4.7 vision on 15 frames per video; and (2) deterministic mechanical-compliance checks on the rendered MP4 — duration, aspect, audio, on-screen typography. These blend into the Arm 3 score with the hallucination component weighted 3×.

Hallucination & Visual-Defect Score (0–10, lower is better)

15 evenly-spaced frames per video sent to Claude Opus 4.7 with the brief's product description as reference. Six categories scored 0–10 (higher = more defects). Pass-rate = share of briefs scoring < 3.0 (production-grade threshold).

Production Polish — deterministic structural quality (0–1)

Four no-LLM measurements on the rendered MP4. Each cell is a 0–1 quality score averaged across every brief in the set. Duration accuracy: ±0.5s of target · Aspect ratio: within 5% · Audio cohesion: contains an audio stream · Text-rendering accuracy: fraction of required on-screen strings detected (OCR + Claude vision).

Conclusion

1. Creatify wins decisively against every competing agent. Arm 1 head-to-head: Pro mode 85% win-rate and Lite mode 76% against the field. The strongest competitor (Luma) wins only 34% of its agent-vs-agent pairs; HeyGen 0%.

2. Creatify dominates raw T2V baselines on ad quality. Arm 2: Pro mode 95% win-rate, Lite mode 89%. The best raw foundation-model baseline lands at 4%.

3. Creatify produces the cleanest video deliverable. Pro mode production-quality score 0.990 · Lite mode 0.965 · best non-Creatify baseline 0.922 (Luma). Pro mode hallucination score 1.90 · Lite mode 2.85 · raw T2V multi-scene 3.65–3.85.

4. Pro mode beats Lite mode on the head-to-head arms. Pro's per-scene QA-review gate adds +9 points of arm-1 win-rate, +7 points of arm-2 win-rate, −0.95 hallucination defects, and +2.5 points of production quality — at ~2× cost and ~2× latency.

5. Creatify delivers a ready-to-ship ad, not a raw clip. Lite at $3/video and Pro at $6/video — each producing a fully assembled branded ad. Full audio cohesion, tight duration match, 89–96% on-screen text rendered correctly vs 15–21% for raw T2V baselines.

The bench is open-source. Every brief, every raw judge verdict, every output video, and every aggregation script is published under Apache 2.0 at github.com/creatify-ai/VABench.

What's next

v2 will scale the benchmark along two axes. First, more briefs covering vertical-specific ad patterns: e-commerce, app install, B2B SaaS, healthcare, education. Second, a closed-loop track that measures the value of multi-turn revision. We are also folding campaign-level signals (CTR, ROAS) into a domain-specific AdReward to drive DPO-style alignment of the underlying I2V models.

Acknowledgements

Creatify Agent and this benchmark stand on the shoulders of a community of researchers working on multi-agent video, video-quality evaluation, ad-effectiveness rubrics, and closed-loop quality assurance. Notable threads we drew on:

• Multi-agent video generation: Mora, VideoDirectorGPT, FilmAgent, StoryAgent, VISTA.
• Video-quality benchmarks: VBench / VBench-2.0; VideoGen-Eval; Artificial Analysis Video Arena.
• Reward modeling for video: VideoReward / Flow-DPO, UnifiedReward.
• Ad-specific rubrics: Springboards Creativity Benchmark, AdsQA, the Pitt Ads Dataset.
• Closed-loop quality assurance: Meta-Harness; Learning Beyond Gradients.