# How I built a music video in Python with Claude Opus 5.5

- Source: https://intervueclub.com/blog/music-video-in-python-claude-opus-5-5
- Author: Dhananjay Aggarwal
- Published: 2026-10-11
- Tags: ai, creative-coding, claude, audio, rendering

Every frame of a 3:30 KR$NA music video, drawn in Python with Claude Opus 5.5: librosa beats, Demucs stems, skia-python and parallel FFmpeg rendering.

Player not loading? [Watch my Vyanjan music video on YouTube](https://www.youtube.com/watch?v=6mTCEq2WdxE).

**TL;DR**

- **Vyanjan** is a 3:30 music video for KR$NA's track where all **12,594 frames** are drawn in Python, built with Claude Opus 5.5.
- The song becomes **60 fps data**: Demucs stems, librosa beats and 707 word events, and every word adds something visible.
- **Every frame is a pure function of time**, which gives parallel rendering, single-frame previews and reproducible output for free.
- Hindi text needs **libraqm shaping**, and the camera and cuts were **measured, not eyeballed**.
- The full engine, data and concept sheets are in **[the Vyanjan Kit](https://topmate.io/dhananjay6561/2343493)**.

**Vyanjan** is a 3 minute 29.9 second music video for KR$NA's 2016 Hindi rap of the same name, and all **12,594 frames** are drawn in Python. I made it with Claude Opus 5.5: the whole video is code, I designed every scene, built the simulations, camera paths and quality checks, and Opus 5.5 wrote the scene code alongside me. The full code, data and concept sheets are in [the Vyanjan Kit](https://topmate.io/dhananjay6561/2343493).

There is no After Effects project, no stock footage and no illustration. Each of the 33 Devanagari consonants gets its own scene, every scene hands over to the next without a cut, and every motion on screen is timed to the song's own audio. This post is the flow I followed, from the song to the final render.

## Vyanjan at a glance

| Fact | Value |
| --- | --- |
| Length | 3 min 29.9 s, 1920 × 1080 at 60 fps |
| Frames | **12,594**, every one drawn by Python |
| Scenes | 33 consonants, plus an intro and a finale |
| Tempo | **99.38 BPM**, 337 detected beats |
| Word events | **707**, each one adds something visible |
| Stack | skia-python, Pillow + libraqm, librosa, Demucs, NumPy, FFmpeg |
| Built with | Claude Opus 5.5, across many sessions |

## What Vyanjan is

**Vyanjan** (व्यंजन) is the Hindi word for consonant, and KR$NA's track runs through the Devanagari consonants in their classical order. That order is a map of the mouth: from the throat (क ख ग घ), through the palate, the curled-back tongue and the teeth, to the lips (प फ ब भ म), then the semivowels, the sibilants and three joined letters. The song skips ङ, ञ and ण, which leaves 33.

Each consonant gets one scene built around a word that starts with it: क is कबूतर (pigeon), घ is घड़ी (clock), ज is जुगनू (firefly). Every scene has the object, a real piece of science, and data panels with real numbers.

One rule came before any code: **the lyrics never appear on screen.** The song belongs to its artist. The video responds to the *timing* of the voice, never its words, and nothing in this post quotes the song either.

## The stack

A music video in Python is a program that turns a time `t` into an image, run once per frame, with the frames piped into an encoder. Python is slow per pixel, so C and C++ libraries do the heavy work and Python decides what to draw.

- **[skia-python](https://github.com/kyamagu/skia-python)** draws every line, fill, blur and composite on a CPU raster.
- **[Pillow](https://pillow.readthedocs.io/en/stable/reference/ImageFont.html) with [libraqm](https://github.com/HOST-Oman/libraqm)** shapes the Devanagari text.
- **[librosa](https://librosa.org/doc/0.11.0/generated/librosa.beat.beat_track.html)** finds the beats and onsets; **[Demucs](https://github.com/facebookresearch/demucs)** splits the song into stems.
- **NumPy and SciPy** run the simulations: fireflies, a vibrating string, orbits, waves.
- **[FFmpeg](https://ffmpeg.org/)** encodes the frames and muxes the audio.

None of it needs a GPU. The whole film renders on a laptop CPU.

## Turning the song into data

Visuals react to music when every animated value reads from numbers computed at the video's frame rate. Vyanjan stores **one value per signal per frame**, 60 a second, so frame 4,560 reads row 4,560 of every track.

**Stems first.** A full mix is a poor signal, because the kick drum and the voice overlap in time and frequency. Demucs separates the track into drums, bass, other and vocals, and each signal comes from its own stem and band: kick, snare and hi-hat from the drums, word onsets from the vocals. Using the vocal stem is what makes the picture follow the rapper rather than the drums.

**Raw for *when*, smoothed for *how much*.** Every signal is stored twice. The raw track decides when something happens; a smoothed copy decides how big it is. The smoothing is one line, applied frame by frame:

```python
v += (x - v) * (attack if x > v else release)
```

Driving motion straight from raw onsets is the usual reason audio-reactive visuals jitter. Triggering on the raw track and moving with the smoothed one gives motion that hits on time and then settles like a physical object.

**Words become events.** Vocal onsets fire at roughly syllable rate, and merging any two closer than 0.11 s turns 741 of them into **707 word events**. The house rule: **every word adds something visible.** A pigeon lands, a weight drops, a lamp lights, a constellation line draws.

**Finding where each letter starts** took two passes. Speech recognition gave a first estimate of the 33 start times, snapped to the nearest beat. It fails in predictable places on this song: retroflex consonants get heard as dental (ट → त), aspirated ones as unaspirated (थ → त). So I checked the starts where it struggled by ear and corrected them. The biggest fix was थ, which moved **2.42 s**, a full bar.

## Every frame is a pure function of time

This is the decision the whole project rests on: **frame `f` depends only on `f`.** The renderer turns `f` into `t = f / 60`, finds the letter playing at that time, and calls that letter's scene function. Nothing may depend on the previous frame: no position nudged each frame, no particle list that grows.

Plenty of scenes need history anyway, and three patterns covered all of them:

- **Closed-form motion.** A value driven by word events is a sum of exact spring responses, one per event, evaluated at `t`. No integration step, so any frame computes on its own.
- **Simulate once, then look up.** The fireflies in ज follow the [Kuramoto model](https://en.wikipedia.org/wiki/Kuramoto_model) and slowly fall into step. The first time the scene needs them, it simulates the whole scene window and stores one state per frame; every frame after that is a lookup.
- **Seed randomness by the frame.** Film grain comes from `random.Random(f)`, so frame 5,000 gets the same grain whichever process draws it.

That one rule paid for itself three times. Any frame renders alone, so review sheets of a dozen stills became the main working loop. Any range renders in parallel. And the output is reproducible: a copy of the engine matched the original pixel for pixel at 11 test frames across the film, apart from one 1/255 difference in a single spectrogram.

![A clock face with the letter घ on it, an x-ray gear train behind the face, and data panels showing a pendulum phase portrait and its length of 0.362 m.](https://intervueclub.com/blog/music-video-in-python-claude-opus-5-5/gha-clock-pendulum-two-beats.webp)

It also turns a lot of the film into arithmetic. The clock in घ swings with a period of exactly two beats, T = 2 × 0.6037 = 1.2075 s, which solves to a pendulum length of **0.362 m**, the length on screen. The heart in ह beats 99.38 times a minute, on the song's tempo.

## Drawing every frame with skia-python

Every frame goes through the same pipeline in the same order: the scene draws its world through its camera, bright parts get a soft bloom, then the panels, captions and HUD are drawn *after* the bloom so text never glows, and finally a vignette and film grain.

The look is deliberately narrow: a near-black background, warm white line art, and orange reserved for whatever is active right now.

![Glowing fireflies gathered into the shape of the letter ज over dark water, with a phase heatmap panel showing their flashes locking into vertical stripes.](https://intervueclub.com/blog/music-video-in-python-claude-opus-5-5/ja-fireflies-kuramoto-sync.webp)

## Drawing Hindi text correctly

Devanagari needs a text shaping engine. Without one, conjuncts like क्ष split into their parts, matras land in the wrong place and the headline (शिरोरेखा) breaks between letters. skia-python is excellent for vector drawing, so the film routes every Devanagari string through Pillow with libraqm instead, then hands the result to skia as a mask it can tint any colour:

```python
font = ImageFont.truetype(path, size, layout_engine=ImageFont.Layout.RAQM)
```

Three things caught me here:

- **On macOS, Pillow has to find libraqm.** Install it with Homebrew and set `DYLD_LIBRARY_PATH=/opt/homebrew/lib`, or Pillow quietly falls back to basic layout.
- **Fonts fail silently.** A font without Devanagari draws boxes, so Devanagari only ever goes through [Mukta](https://fonts.google.com/specimen/Mukta).
- **Test every glyph you show.** A missing glyph doesn't raise an error; it just draws a box in frame 9,000.

## Measuring what "smooth" means

A camera that looks fine on a contact sheet can feel terrible in motion, because what you feel watching is speed and acceleration, not position. So every camera path was measured before it rendered.

That caught problems the eye missed. In द, a camera that jumped to each newly lit lamp peaked near **14,000 px/s²** of acceleration; rebuilt as one even climb, it peaks at **542**. In ध, a 1,320 °/s whip came down to 83 °/s.

The same idea checks the cuts. Vyanjan has no hard cuts, only handovers: raindrops glow into fireflies, a flaming arrow becomes a shooting star. A **cut ratio** compares how much the picture changes across a handover with an ordinary frame step. At 1.0 the seam is invisible; above **1.3** it goes back for rework.

![A temple staircase of oil lamps in line art, an orange ember on one of the steps, and the stair lines brightening with distance from the flame.](https://intervueclub.com/blog/music-video-in-python-claude-opus-5-5/da-lamps-staircase-camera.webp)

![An archery target face with the letter ध projected onto it, arrows in the letter, and a bow in the foreground.](https://intervueclub.com/blog/music-video-in-python-claude-opus-5-5/dha-archery-target-letter.webp)

## Rendering 12,594 frames in parallel

Because frames are independent, the film splits into ranges with no coordination. Each process renders its range and pipes raw frames straight into its own FFmpeg (libx264), then FFmpeg's concat joins the parts without re-encoding and the song is muxed once at the end.

Nothing touches the disk as images. A frame is 1920 × 1080 × 4 bytes, so the film is about **104 GB of raw pixels**, all streamed. The heaviest shots, like the heart and the finale's spiral, take up to about 0.5 s a frame; the full film averages about 0.12 s a frame on an M1. The practical limit on parallel jobs is memory, not cores.

The veena in व is the one expensive simulation, so its results are cached, and the cache key includes a hash of the simulator's own source file. Change one constant and the next render re-simulates on its own; there's no "remember to clear the cache" step.

![A Saraswati veena in line art with a simulated string drawn as its vibration envelope, an inset showing the string grazing the curved jivari bridge, and a live spectrum panel.](https://intervueclub.com/blog/music-video-in-python-claude-opus-5-5/va-veena-string-simulation.webp)

![A 3D line-art heart with chambers, valves and great vessels, next to an ECG monitor trace and conduction-stage labels.](https://intervueclub.com/blog/music-video-in-python-claude-opus-5-5/ha-heart-beating-on-tempo.webp)

## Shipping the engine without the song

The kit contains no audio, no stems and no lyrics; a buyer brings their own copy. But different encodes of the same recording start a few milliseconds apart, and all the timing data is relative to the exact file I analysed. A 50 ms offset is three frames, enough to put every word visibly off the voice.

So the kit ships a **fingerprint**: a loudness envelope at 100 values a second that can't be played or turned back into audio. `prepare_audio.py` cross-correlates the buyer's copy against it to find the offset, refines it below one envelope frame, and refuses a copy that doesn't match. In testing it recovered a 0.5 s delay as +500.2 ms, and rejected a copy sped up by 3 %, which would have drifted 6 seconds off the voice by the end.

## Building it with Claude Opus 5.5

[Claude Opus 5.5](https://www.anthropic.com/claude-opus-5-5) wrote the scene code across many sessions. I built the simulations, camera paths and quality checks, decided what the film is, approved each scene, and did the parts that need ears and eyes.

| Part of the work | Who did it |
| --- | --- |
| The rules (no lyrics, line art on black, every number real, every word visible) | Me |
| Each letter's concept (beat table, bridges, science, camera plan) | Claude Opus 5.5 proposed it; I changed and approved it |
| Scene code | Claude Opus 5.5 |
| Simulations, camera paths, quality checks | Me |
| Correcting the letter starts the recogniser got wrong | Me, by ear |
| Whether a scene was finished | Me |

Three things made it work:

- **Rules before scenes.** No lyrics on screen, every word adds something visible, orange only for what's active, every number on screen computed or cited. They went into one house-style document every scene was held to.
- **A concept before code.** Every letter started as a written plan: exact word times, what each word adds, the bridge in and out, the camera. Agreeing on that first wasted far less work than building and revising.
- **A memory that outlives the session.** Each session starts fresh, so the project kept a handoff document read at the start of every one: the commands, the rules, and a status table for every letter.

Numbers settled the reviews. "It looks smooth" isn't evidence from anyone, while a peak of 542 px/s² is the same number for both of us. Ears and taste stayed with me: the timing fixes came from listening, and so did the calls on whether a scene read as the film or as a children's textbook.

![All 33 consonants arranged along one continuous spiral that nests into itself, an Escher-style Droste effect, in white and orange on black.](https://intervueclub.com/blog/music-video-in-python-claude-opus-5-5/jna-droste-spiral-finale.webp)

## Key takeaways

- **Make every frame a pure function of time from day one.** Retrofitting it later is a rewrite.
- **Keep a raw and a smoothed copy of every audio signal.** Raw for timing, smoothed for motion.
- **Drive visuals from separated stems,** not the full mix, so the picture follows the voice.
- **Measure what you'd otherwise judge by eye:** acceleration, pan rate, cut ratio.
- **Put the code in the cache key,** so a changed simulator can never serve stale results.
- **Shape Devanagari with raqm and test every glyph;** missing glyphs fail silently.
- **Give an AI pair rules, written plans and a memory.** That's what turned many sessions into one consistent film.

The arithmetic here (frames times bytes, worst-case render time) is the same kind the [back-of-the-envelope math](/course/back-of-the-envelope-math) lesson practises.

## Frequently asked questions

### Can you make a music video with Python?

Yes. Vyanjan is 12,594 frames at 1920 × 1080 and 60 fps, every one drawn with skia-python, NumPy and SciPy and piped into FFmpeg. No animation software was used.

### How do you sync animation to music in Python?

Compute features at your video frame rate, detect beats with librosa and onsets per frequency band, trigger motion on the raw onsets and size it with a smoothed copy.

### Does Pillow support Hindi and Devanagari text?

Yes, when it's built with libraqm and you load the font with `layout_engine=ImageFont.Layout.RAQM`.

### Is the song included in the Vyanjan Kit?

No. [The kit](https://topmate.io/dhananjay6561/2343493) has the renderer, the data, the docs and the concept sheets, but no audio, stems or lyrics. You bring your own copy and `prepare_audio.py` aligns it.

## Get the Vyanjan Kit

If you want to render Vyanjan yourself or reuse its engine in your own work, **[the Vyanjan Kit](https://topmate.io/dhananjay6561/2343493)** has everything this post leaves out:

- **The renderer:** 11,220 lines of Python across 41 modules, all 33 scenes plus the intro and finale
- **The audio data:** 60 fps drum and vocal tracks, the 337-beat grid, 707 word events and every letter's start time, as NumPy and CSV
- **The docs:** a 16,800-word reference covering every scene's science, numbers and camera, plus the quality checks
- **The concept sheets:** beat tables, bridges between letters and camera plans
- **The tools:** `prepare_audio.py` to align your copy of the song, and the analysis pipeline for songs you have the rights to

No audio is included; you bring your own copy of the song. **[Get the Vyanjan Kit →](https://topmate.io/dhananjay6561/2343493)**

*Song: "Vyanjan" by KR$NA (2016). All rights to the song, its recording and its lyrics belong to the artist and label. The video is an independent visual work, not affiliated with or endorsed by the artist, and no lyrics appear in it or in this post.*