| name | lyric-video-director |
|---|---|
| description | Use when creating a stylized lyric video from a song file, lyrics, and rough lyric timings. Directs the complete workflow: normalize timings, analyze musical gaps and song structure, establish a visual bible, use GPT Image 2 to design creative final typography compositions and reference assets, animate words and graphic elements with Gemini Omni Flash reference-to-video, preserve exact lyrics through adaptive text-safety, assemble against the untouched master song, and verify synchronization, legibility, spelling, coverage, and audiovisual quality. Also use for kinetic-typography and instrumental visualizer sections. |
| version | 1.0.0 |
| author | Hermes Agent |
| license | MIT |
| platforms | ["linux", "macos", "windows"] |
| metadata | {"hermes": {"tags": ["lyric-video", "music-video", "typography", "gpt-image-2", "gemini-omni", "ffmpeg", "audio-analysis"], "related_skills": ["songsee", "songwriting-and-ai-music"]}} |
Lyric Video Director
Overview
Create a lyric video as a coherent piece of music direction, not a slideshow of unrelated captions. The lyric must remain correct, the edit must serve the song, and every flourish must earn its place.
The expected minimum input is:
- A song file.
- Lyrics.
- Rough start/end timing for each lyric line.
The pipeline analyzes the entire song, accounts for every instrumental gap, locks a visual language, uses GPT Image 2 to design creative final typography compositions and reusable reference assets, animates words and graphic systems with Gemini Omni Flash reference-to-video, discards generated audio, and composites the visuals against the user's untouched song.
Default visual grammar for typography-led work: the lyric is the protagonist. Build rapid composition resets, word-object metaphors, mixed-media graphic systems, spatial typography, illustration, and recurring visual motifs. Do not default to portrait-led cinematic scenes with captions laid beside them.
Default generation route: use mcp__fal_omni__omni_reference_to_video with an ordered set of uploaded GPT Image 2 references and bind them explicitly as <IMAGE_REF_0>, <IMAGE_REF_1>, and so on. Use image-to-video only when the user asks for a single-keyframe animation or when a shot genuinely requires one fixed composition.
Default text philosophy: creative generated words should actually animate, transform, collide, stack, and become objects. Exact-text verification remains mandatory; use deterministic fallback only for failed words or shots, not as an excuse to turn the whole video into static captions.
Read linked references only when their phase is reached:
references/audio-and-timeline.md— analysis, gap classification, timing refinement.references/instrumental-narrative-direction.md— wordless narrative arcs, motif accounting, audio-to-action mapping, and instrumental QC gates.references/non-lyric-retrofit-workflow.md— surgically replace rejected gaps, solos, transitions, and outros while preserving approved lyric sections and audio.references/art-direction-and-prompts.md— visual bible and GPT Image 2 composition prompts.references/kinetic-typography-reference-video.md— typography-first visual grammar and Omni reference-to-video prompting.references/full-song-production-qc.md— segmented long-form production, per-shot provenance, creative-parity gates, and localized repair patterns.references/typography-and-qc.md— text choreography, OCR checks, fallback logic.references/worked-examples.md— examples across genres and gap types.templates/project.yaml— canonical project manifest.
When to Use
Use for:
- Full lyric videos from audio + timed lyrics.
- Short-form lyric clips for Reels, TikTok, Shorts, or social posts.
- Music visualizers where instrumental sections must connect lyric scenes.
- Repeated chorus/hook treatments that evolve coherently.
- Songs with rough line timings that need musically intelligent refinement.
Do not use for:
- Plain subtitle burning with no creative direction; use deterministic caption tools.
- Karaoke scoring or pitch detection unless explicitly added to scope.
- Replacing, remixing, time-stretching, or regenerating the supplied song.
- Creating imagery before the song and complete timeline have been analyzed.
Non-Negotiable Rules
- Master audio is sacred. Use the supplied song as the sole final soundtrack. Mute/discard all Gemini-generated audio. Do not time-stretch, remix, normalize destructively, or regenerate it unless asked.
- No unexplained timeline holes. Cover the entire duration from first frame to last frame, including intro, breaths, solos, fills, pauses, and outro.
- Music drives shot boundaries. Lyric lines inform the timeline but do not automatically equal clips.
- Visual bible before bulk generation. Never generate a full song as unrelated one-off prompts.
- Exact lyric text wins. Never ship misspelled, duplicated, mutated, censored, or missing words because a shot looks cool.
- No silent censorship. Ask or infer from explicit user direction whether lyrics are verbatim, clean, or dual-export.
- Tracer before scale. For long or expensive projects, validate one verse line, one high-energy line, and one instrumental gap before bulk generation.
- Bounded retries. Default to two image corrections and two video retries per stubborn shot, then use the text-safe fallback.
- Verify the rendered artifact. An MP4 existing is not completion.
- Do not substitute models silently. If GPT Image 2 or Gemini Omni Flash is unavailable, report the blocker and ask before changing the creative stack.
Intake Contract
Accept loose human formatting such as:
Twinkle twinkle little star - 0:00 - 0:05
How I wonder what you are - 0:05-0:10
Bright above the sky so high - 0:13-0:18
Like a diamond in the sky - 0:18 - 0:23
Normalize internally to seconds with millisecond precision. Preserve the exact supplied lyric string separately from any display line-breaking.
Required
- Song path or uploaded song asset.
- Exact lyrics.
- Rough line-level timings.
Ask only when materially missing
- Aspect ratio/platform target; default to
16:9for a general master. - Visual references or requested theme; otherwise derive 2–3 directions from the song.
- Explicit/clean policy if relevant.
- Title/artist/credits requirements.
- Desired resolution; default to 1080p delivery.
Do not interrogate the user for choices the song analysis can resolve.
Normalize and validate
Detect and report:
- Reversed or impossible ranges.
- Overlaps, including intentional duet/ad-lib overlaps.
- Timestamps outside the song.
- Duplicated or omitted lines.
- Dense lines with inadequate reading time.
- Unsupported characters or script/font risks.
- Gaps and trailing space after the final lyric.
Represent simultaneous voices explicitly:
[Lead] Main lyric | 00:03.000 --> 00:07.000
[Backing] Secondary phrase | 00:05.200 --> 00:07.000
Completion criterion: every supplied lyric exists exactly once in the normalized manifest unless repetition or overlap is intentional and labeled.
End-to-End Workflow
1. Inspect prerequisites and media
- Confirm the audio file exists and is decodable.
- Use
ffprobeto collect exact duration, codec, channels, sample rate, and start time. - Confirm
ffmpegis available for final assembly. - Confirm access to GPT Image 2 and the FAL Gemini Omni tools.
- Create a project directory with
source/,analysis/,art/,clips/,qc/, andoutput/.
Completion criterion: the song is decodable, duration is known, creative backends are available, and no generation has started prematurely.
2. Analyze the song
Read references/audio-and-timeline.md. Create:
- Waveform/loudness map.
- Onset/transient map.
- Beat/downbeat estimate when reliable.
- Spectral and harmonic/percussive observations.
- Likely section boundaries and energy arc.
- Descriptions for every non-lyric interval.
Use songsee for visual evidence when available:
songsee song.wav --viz spectrogram,mel,chroma,hpss,selfsim,loudness,tempogram,flux -o analysis/song-map.png
Do not label every gap “instrumental.” Describe its function: drum fill, held chord, breath, rise, drop, solo, decay, silence, or transition.
Completion criterion: the analysis explains what the listener hears across the entire song, especially every lyric-free interval.
3. Build a coverage-complete timeline
Partition [0, song_duration] into contiguous coverage intervals. Each interval records:
- Start/end and duration.
- Type:
intro,lyric,overlap,instrumental,transition,outro. - Musical event and energy.
- Lyric and voice, if any.
- Visual purpose.
- Proposed shot boundaries and internal cue points.
- Transition in/out.
- Typography mode.
A lyric interval may contain several shots; several short lyrics may share an evolving shot. Respect Omni's supported clip duration and split long intervals on musical events, not arbitrary equal chunks.
Completion criterion: intervals begin at zero, end at exact song duration, and contain no positive gap or accidental overlap.
4. Direct the visual concept
Read references/art-direction-and-prompts.md. Produce 2–3 concise concepts derived from:
- Genre, instrumentation, texture, tempo, and emotional arc.
- Lyric imagery and recurring symbols.
- Platform/aspect ratio.
- Technical need for readable typography.
Choose or ask the user to choose only when the visual directions have genuinely different trade-offs. Then lock a visual bible containing:
- One-sentence concept/logline.
- Emotional and energy progression.
- Palette and forbidden colors.
- Materials, world, lighting, and depth strategy.
- Typography personality, hierarchy, maximum line count, and safe margins.
- Motion grammar and prohibited motion.
- Camera grammar.
- Transition family.
- Recurring motifs and how repeated hooks evolve.
Completion criterion: another artist could create three consistent scenes from the visual bible without guessing the style.
5. Make a tracer sequence
For songs longer than a short test, generate only:
- One representative verse kinetic-typography setup.
- One chorus/high-energy composition with a semantic word transformation.
- One instrumental-gap graphic composition.
- GPT Image 2 start/final references and their Omni reference-to-video animations.
- A 10–20 second assembled preview using the actual song excerpt.
Evaluate coherence, readability, motion, and transition potential before bulk generation. If the user is present, show the tracer; if autonomous generation was explicitly requested, inspect it rigorously and proceed only when it passes.
Completion criterion: the tracer proves the style survives both image generation and animation.
6. Generate motion-ready lyric artwork
Use GPT Image 2. Each prompt must include:
- Exact lyric in a clearly delimited block.
- “Render exactly this text; no additional words.”
- Visual-bible constraints.
- Layout, safe margins, text size, and quiet contrast zone.
- Foreground/midground/background depth layers.
- Which elements should later move.
- A transition affordance near the intended exit edge.
- No watermark, logo, border, signature, or incidental writing.
Generate instrumental frames from musical function, not generic ambience. Reuse motifs and reference images to maintain continuity.
Inspect every image before animation. Correct spelling and composition first; animation cannot rescue a bad keyframe.
Completion criterion: the keyframe is visually coherent, mobile-legible, contains no unwanted text, and has usable motion layers.
7. Animate with Gemini Omni Flash reference-to-video
Upload every GPT Image 2 composition and reusable visual element with mcp__fal_omni__upload_asset, then use mcp__fal_omni__omni_reference_to_video. Pass an ordered image_urls array and bind every source explicitly in the prompt as <IMAGE_REF_0>, <IMAGE_REF_1>, and so on.
Reference sets should normally include:
- A start composition showing the initial lyric layout.
- A final composition showing the intended payoff or transformed layout.
- Optional element references for recurring icons, materials, type treatments, characters, or transition motifs.
GPT Image 2 should design the creative final states. Omni's job is to perform the transformation between and through those references: reveal, stack, collide, erase, stretch, fold, replace, or spatially reorganize words according to the music.
Prompt with bounded choreography:
- Name every
<IMAGE_REF_N>and explain its role. - State the exact lyric strings that may appear.
- Specify which words enter at which timestamp and how each behaves.
- Describe the relationship between words and illustrated objects.
- Specify the intended final composition and transition-out.
- Forbid extra writing, misspellings, duplicate words, and unrelated scene invention.
- End with “Keep every unmentioned design element consistent with the references.”
Use omni_image_to_video only when a shot intentionally has one fixed keyframe and no meaningful reference-driven transformation. Do not default to it for kinetic typography.
Do not ask Gemini to recreate the supplied song. Generated native audio is disposable.
For continuity, reuse style sheets, recurring motifs, icon families, paper/material libraries, and matching type treatments across reference sets. Prefer bold composition resets—wipe, fold, stamp, tear, erase, zoom-through, object collision—over portrait continuity.
Completion criterion: the words visibly perform the lyric, the final frame resolves toward the GPT Image 2 target composition, and every reference used in the prompt is accounted for.
8. Apply adaptive text safety
Read references/typography-and-qc.md.
For each lyric clip:
- Sample beginning, quarter, midpoint, three-quarter, and end frames.
- Check exact spelling and legibility.
- If reliable, retain generative typography.
- If unstable, retry with less motion around text.
- If still unstable, cover/reconstruct the lyric region and render exact typography deterministically while preserving the animated world.
The user may request a deliberately fully generative alternate, but the master remains text-safe unless they explicitly waive correctness.
Completion criterion: every visible lyric matches the canonical lyric string for its active interval.
9. Assemble to the song
Use deterministic compositing to:
- Trim clips to timeline intervals.
- Add modest speed adjustment only to visuals when needed.
- Create transitions on musical cues.
- Render exact overlays/highlights.
- Concatenate at a single resolution, frame rate, pixel format, and colorspace.
- Remove every generated clip's audio.
- Mux the original supplied song as the final soundtrack.
- Preserve song duration exactly, allowing only normal container/frame rounding.
Avoid transition spam. Cuts, holds, and silence are legitimate design choices.
Completion criterion: the video covers the entire song, ends with the song, and contains only the intended master audio.
10. Quality-control and deliver
Run the verification checklist below. Inspect representative frames and watch the final output with sound. Produce:
- Final master MP4.
- Optional platform exports.
- Timeline/manifest.
- Visual bible.
- A concise note of any adaptive fallbacks used.
Completion criterion: the final video passes technical, textual, musical, and visual checks—not merely file existence.
Instrumental Gap Playbook
Read references/instrumental-narrative-direction.md before directing any lyric-free interval longer than three seconds.
When the user approves the lyric sections but rejects only the instrumental material, do not rebuild the song. Lock the approved master, map only the rejected non-lyric intervals, and follow references/non-lyric-retrofit-workflow.md to generate silent exact-frame replacements, preserve the approved audio, and verify retained lyric frames independently.
Instrumentals are wordless narrative sequences, not attractive filler. Each substantial gap must inherit a dramatic question from the preceding lyric, develop a visual mechanism through audible musical changes, and causally create the next lyric composition.
Classify by duration and musical function:
| Gap | Default treatment |
|---|---|
<1.5s | Keep inside surrounding lyric choreography: finish a tear, stamp, fold, blink, rebound, collapse, or clear. Do not introduce a new scene. |
1.5–3s | Compact visual answer to the outgoing lyric and a causal setup for the next composition. |
3–8s | Complete wordless mini-shot with inherited question, setup, development, payoff, and handoff. |
>8s | Multi-shot narrative section with several GPT Image 2 references, clear internal phases, musical escalation, changed motifs, and a full layout refresh. |
| Outro | Purposeful closure across measured phases: resolve, strip systems away, return to a changed opening motif, or decay to a final symbol/state. |
For every interval over three seconds, record:
- inherited dramatic question;
- audible drivers and cue times;
- visual mechanism;
- setup, development, escalation/reversal, payoff;
- established motifs reused and how their meaning changes;
- next layout created by the final action.
Reject an instrumental sequence if it could be moved elsewhere in the song unchanged, loops or stretches one clip, lacks consequences, ignores major musical changes, introduces disposable unrelated imagery, or fails to explain why the next visual world appears.
Never put the next lyric on screen early merely to fill a gap. Background anticipation is allowed; canonical lyric visibility follows the chosen onset policy.
Repeated Sections
Repeated hooks should recur recognizably but evolve. Do not keep one layout system for the entire song. At major section boundaries, perform a full visual refresh: clear the previous text/world completely and introduce a new compositional grammar while retaining a small set of master DNA such as palette, paper texture, type families, or recurring motifs.
- Verse 1: establish the first layout system.
- Chorus 1: refresh into a bolder hero composition.
- Instrumental/solo: full text-free reset with its own graphic world.
- Verse 2: introduce a clearly different layout family, not merely recolored Verse 1.
- Chorus 2: recall the chorus motif with altered spatial organization.
- Bridge: full refresh suited to the emotional turn.
- Final chorus: culmination, altered material state, expanded palette, or accumulated visual history.
- Outro: strip systems away or introduce a final minimal grammar.
A full refresh should feel intentional rather than random: use one transition that completely clears the outgoing composition—shred, stamp, fold, erase, collapse, zoom-through, page replacement—then land in a visibly different grid, depth model, or illustration language. Continuity comes from recurring motifs and palette, not from keeping every shot visually identical.
Reading and Timing Rules
Track four different events:
- Visual anticipation — scene motion may begin before the vocal.
- Lyric onset — canonical lyric becomes readable.
- Emphasis cue — active phrase/word reacts.
- Release tail — composition remains briefly after the vocal.
Estimate reading density. If a line is too dense:
- Split it into phrase reveals.
- Increase stable reading time.
- Reduce ornamentation and motion.
- Use a larger composition with fewer concurrent elements.
- Ask for corrected timing only when analysis cannot resolve it safely.
Do not fake word-level precision when only line-level timing is known. Mark inferred timings and confidence; use phrase-level treatment when confidence is weak.
Common Pitfalls
- One image per line. This creates a slideshow. Group or split shots according to musical structure.
- Generating before audio analysis. Instrumental gaps and transitions become arbitrary.
- Trusting first-frame spelling. Video text can mutate later; inspect multiple frames.
- Prompting “animate cinematically.” Vague motion produces chaos. Use timestamped choreography.
- Letting every object move. Motion hierarchy is necessary; protect quiet zones around lyrics.
- Using generated audio. Always discard it and mux the original song.
- Overusing transitions. A clean cut on a downbeat often beats another morph.
- Forcing final-frame continuity. It compounds artifacts. Use motif continuity and clean compositional resets.
- Making lyrics readable only at full resolution. Inspect at mobile display size.
- Silent censorship or rewriting. Preserve canonical text and user intent.
- Infinite retries. Use bounded retries and deterministic fallback.
- Ignoring outro space. The final gap is an ending, not leftover duration.
- Passing a local path to FAL image editing. FAL's edit endpoint must fetch source images by public URL. Upload local keyframes first (for example with
mcp__fal_omni__upload_asset) and pass the returned URL asimage_url. - Assuming FFmpeg has
drawtext. Homebrew/minimal FFmpeg builds may omit libfreetype. Check filter availability; when absent, render the exact lyric as an RGBA PNG with Pillow and composite it using FFmpeg'soverlayfilter. - Assuming GPT Image 2 returns native 16:9 through FAL. The FAL catalog currently maps GPT Image 2 landscape generation to a 4:3 preset to satisfy minimum pixel requirements. Compose inside a protected 16:9 center-crop region, then crop or let the video stage render 16:9 deliberately.
- Overloading one reference-to-video shot. Asking Omni to assemble a long lyric and transform into the next shot can duplicate short words such as
IorCAN. Split the lyric animation and transition into separate calls, reduce simultaneous transformations, and make each final composition an explicit hold. - Treating one invented glyph as total failure. If reference-to-video animates the words correctly but invents one static quote or mark, repair only that tracked region with a small inpaint/overlay. Do not flatten all kinetic typography into static subtitles.
- Using
image_to_videoby habit. For typography-led work, default toomni_reference_to_videowith GPT Image 2 start/final compositions. Use image-to-video only when one fixed keyframe is genuinely the intended shot. - Applying the text-safe fallback globally. Deterministic overlays are permitted only as small, local repairs to an otherwise approved kinetic shot. Never render every canonical line as a static PIL/FFmpeg text card over looped Omni backgrounds. That changes the product into a templated lyric overlay and fails the core brief.
- Confusing section backgrounds with final shots. A handful of Omni motion worlds looped across sections do not constitute a complete reference-to-video lyric video. Each lyric group needs designed GPT Image 2 opening/final compositions and word choreography, or an explicitly approved text-free instrumental shot.
- Technical QC without creative-parity QC. Duration, codecs, black-frame tests, and timeline coverage cannot approve the video. Before full assembly, compare sampled frames and motion from every section against the approved tracer. Fail the render if typography is mostly static, layouts repeat as templates, words do not physically perform meaning, or transitions merely fade overlays.
- Self-reporting compliance without artifact evidence. Claims such as “GPT Image 2 compositions animated through Omni” must be backed by a per-shot manifest mapping each interval to actual GPT references and Omni request/output IDs. If the final compositor replaces those with generic overlays, the report must say so and the deliverable must fail.
- Rebuilding approved lyric sections to fix instrumentals. This increases cost and creates avoidable regressions. Lock the accepted master and replace only mapped non-lyric frame ranges; verify retained lyric samples against the baseline.
- Using
-shortestwhen stream-copying the approved soundtrack. A constant-frame-rate video can end a few milliseconds before the complete audio stream.-shortestmay silently truncate the approved audio tail and change its decoded hash. Preserve the full audio stream and allow the container to follow its duration when the difference is below one video frame.
Verification Checklist
Input and timeline
- Audio decodes and exact duration is recorded.
- Every supplied lyric is represented exactly.
- Timeline begins at
0and ends at song duration. - No unexplained gaps or accidental overlaps remain.
- Instrumental intervals have musically grounded descriptions.
Art direction
- Visual bible is locked before bulk generation.
- Palette, materials, typography, camera, and transitions are coherent.
- Repeated sections recur and evolve intentionally.
- Instrumental shots connect surrounding sections.
Typography
- Every lyric is correctly spelled and uncensored unless requested.
- Text is readable at mobile size.
- Stable reading windows are adequate.
- Sampled frames contain no mutation, duplication, or unwanted writing.
- Any failed generative typography uses the text-safe fallback.
Video and audio
- No black frames or accidental freezes.
- Clip boundaries align with musical cues.
- All Gemini-generated audio is absent.
- Original song is the sole soundtrack.
- Output duration matches the song within frame/container tolerance.
- Resolution, frame rate, colorspace, and aspect ratio are consistent.
Final artifact
- Final MP4 opens and plays to completion.
- A human watch-through with sound has been completed.
- Timeline manifest and visual bible are saved alongside the output.
- Final master path is reported to the user.