← Saeyohn Mahendran

Manim Director

An agentic studio that turns one markdown file into a narrated math video, rendered with Manim Community. Independent project, not affiliated with Manim.

Saeyohn MahendranJul 2026 · 4 days
StackTauri v2 · Svelte 5 · Python · Manim · ffmpeg
VoiceKokoro · Chatterbox (MLX) · faster-whisper
AgentsClaude Code · MCP · Express hub

TL;DRA director agent plans each lesson scene by scene, then a coding agent writes the Manim. I review diffs instead of animating by hand, and everything except the drafting runs on my machine.

From the deflection-theory lesson: the cable-slice force balance, drafted by an agent, rendered by the pipeline, narrated locally.

Background

I take a lot of math notes, and I wanted to turn them into videos where the visuals actually compute things on screen. Doing that by hand in Manim costs hours for every minute of finished video, so I built a studio around four decisions:

Architecture

The system has three parts.

One rule holds them together: files are the API. Python is the only thing that parses lesson.md. The app never interprets lesson content itself; it calls studio state and renders the JSON it gets back. Agents edit files, and a filesystem watcher refreshes the app. That single constraint is why the agents, the CLI, and the UI never disagree about what a lesson is.

lesson.md → parse → render (manim) ∥ narrate (tts) → assemble (ffmpeg) → lesson.mp4

The pipeline

Parse

A lesson is ## sections with {#id} anchors. Prose is the narration script, verbatim. Visuals are fenced blocks naming a scene and a template:

## The dead load sets the tension {#s03}

Time to put numbers on the bridge itself. Under its own uniform
weight, the cable settles into a parabola:
$y(x) = \frac{4f}{L^2}\,x\,(L - x)$ ...

```animation scene=s03_deadload template=equation_morph duration=50s
{ "beat_offset": 1, "steps": ["y(x) = ...", "y'' = ...", "H_w = ..."] }
```

Ten parameterized Manim templates (vector plots, grid shears, equation morphs) cover most scenes; template=freeform is hand-written Manim for everything else; template=placeholder is a slot carrying only a prompt, for script-first drafting.

Render

Every scene is content-addressed. The hash covers the generated Manim source, the template class source, referenced asset bytes, and the narration beats baked into the scene. Unchanged scenes never re-render:

h = scene_hash(source, template_cls,
               asset_fingerprint(lesson_dir, block.params))
if manifest.is_current(block.scene, h) and entry["quality"] == quality:
    click.echo(f"  {block.scene}: current ({h}), skipping")

Render quality is sticky per lesson (majority of the manifest). That rule exists because an agent once rendered two scenes at 480p into my 1080p lesson and every build died at an assemble gate for an hour.

Assemble

ffmpeg pads each scene to its section's audio (hold last frame, pad silence), loudnorms, and concatenates. Two hard gates exist because of real incidents: sections with mismatched resolutions refuse to assemble, and the whole build takes an flock on the lesson:

@contextmanager
def lesson_lock(lesson_dir):
    f = open(lesson_dir / "renders" / ".studio.lock", "w")
    try:
        fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except OSError:
        raise LessonBusy("another studio process is already building this lesson")

One night the app, a chat agent, and a CLI build all assembled the same lesson at once. They interleaved writes into the shared intermediates and produced mp4s with no moov atom. One writer per lesson now; the second gets a clean error.

The agents

The agents became reliable through three habits: a file contract they can't drift from, tools where I would otherwise have written instructions, and reading their transcripts to see what they actually did.

The queue contract

Every lesson folder has a requests.jsonl. Humans (via the app) append requests; agents work them:

{"ts":"2026-07-11T16:20:00Z","type":"draft-lesson",
 "note":"Draft the full lesson \"The RC circuit: deriving the time
 constant\". Brief: ..."}

The app spawns a drafter as a plain claude -p process with a fenced toolset:

Command::new("claude").args([
    "-p", prompt, "--output-format", "stream-json", "--verbose",
    "--permission-mode", "acceptEdits",
    "--allowedTools", "Edit", "Write", "Read", "Glob", "Grep",
    "Bash(uv run studio *)", "Bash(studio *)",
    "--max-turns", "40",
]).current_dir(&lesson_dir)

stream-json means every agent step is a line the app can parse: the UI shows a live point-to-point trail (tool, target, thought), a ticking clock, and a "quiet Ns" indicator when the model thinks. Progress persists to .agent.json in the lesson folder, so reopening a lesson shows what its last run did.

The agent panel mid-draft: a dot timeline of Read/Write/Bash steps with thoughts between them
The agent panel mid-draft. Every tool step and thought lands as a dot on the trail, and the run cost and turn count arrive with the final event.

Chat, two scopes

The chat rail is claude -p --resume <session> per turn, one continuous session per lesson, cwd = the lesson folder. A second scope, Studio, points the same machinery at the workspace root with a different system prompt, so "make me a lesson on Laplace transforms from my notes at ~/..." scaffolds, drafts, and files a new lesson into the curriculum. Messages sent while a turn runs queue up and fire automatically; attachments are just file paths the agent reads.

MCP and the delta contract

The hub exposes 17 MCP tools so external Claude sessions can operate the studio. Script editing follows a read-once + patch contract: get_script_model returns the parsed lesson with line spans and a rev (sha256 of the file, truncated); every mutation returns a structured delta against that rev. Agents stop round-tripping whole files, and concurrent edits fail loudly instead of silently clobbering.

Director discipline

Output quality came from a written authoring bar in each lesson's CLAUDE.md, distilled from my actual rejections. The load-bearing rules:

Measuring agents

The written rules only went so far, so I read the transcripts. Across four drafts, every agent burned 5–8 tool calls re-discovering the animation template registry by reading pipeline source, plus 3–5 blocked shell commands. The root cause was my own docs, which told agents to run a python -c one-liner that the permission gate blocks. Instead of rewording the docs I added a tool: studio templates prints every schema in one call.

Then I re-ran a controlled draft (a derivation-heavy brief designed to tempt the old failure modes):

MetricBefore (4-draft baseline)After (one controlled retest)
Tool calls per draft35–6020
Template-discovery calls5–81
Blocked commands3–50
Cost per draft$8–12 at API prices$4.45
Derivation fragmentationequation-card scenesone 6-step morph

Model: Claude Fable 5 via Claude Code (one baseline draft ran Claude Opus 4.8). Dollar figures are Anthropic API list prices as reported by Claude Code's per-run cost accounting, and notional when the agent runs on a subscription login.

The retest also exposed a loophole in the rules themselves: agents kept reaching for the math-only morph template, which structurally can't dock a visual next to the equations. That became another guide fix, and the loop repeats from there.

The controlled retest's output: the RC-circuit derivation as one accumulating six-step morph, drafted, validated, and rendered without a human touching the file.

Voice

Math-to-speech

Prose keeps $...$ TeX for display; the voice must never say "dollar q dollar". Math spans convert to words at synthesis time only:

def _speak_tex(tex):
    s = tex
    while _FRAC.search(s):                      # innermost-first
        s = _FRAC.sub(lambda m: f" {m.group(1)} over {m.group(2)} ", s)
    s = s.replace("''", " double prime ")
    s = re.sub(r"\^\s*\{?\s*2\s*\}?", " squared ", s)
    ...
# "$H_w = \frac{q L^2}{8 f}$"  →  "H w equals q L squared over 8 f"

Display text keeps the TeX (captions render KaTeX); only the audio changes. The narration cache hashes the speakable form, so exactly the math-bearing sections re-synthesize.

Paragraph prosody

The voice sounded robotic on long sentences, and the cause was in my own pipeline. I was synthesizing per sentence to get exact caption timings, so the model restarted its intonation contour every sentence, and the silence-trimmed WAVs were butt-joined with zero gap. The fix was to invert it: synthesize each prose chunk in one TTS pass, then recover the per-sentence beats by aligning faster-whisper word timestamps against the known text with a Needleman–Wunsch pass over normalized words. Transcript drift ("2.47" vs "two point four seven") survives because the surrounding words anchor the alignment, and a proportional split is the deterministic fallback, which also keeps tests hermetic.

Engines and cloning

Kokoro-82M is the default engine: faster than real-time, 54 voices, phonemized by misaki (dictionary-first G2P) after measuring that raw espeak reads "2.47" as "two, forty-seven". Chatterbox (0.5B, MIT, via mlx-audio) is the quality engine at roughly real-time; it zero-shot clones a reference WAV, so any Kokoro voice, or 15 seconds of me reading a script in Settings, becomes the narrator. Engine choice was settled by a blind A/B: same sections, shuffled a/b pairs, answer key sealed until after picking. The pair below is one of the test sections:

Voice A
Voice B. One is Kokoro, one is a Chatterbox clone of the same character. Listen blind, then decide.

The app

The full workspace: curriculum tree, script document view, program monitor, filmstrip timeline
The workspace on the matmul lesson: curriculum tree, script document view, 16:9 program monitor, animation deck, inspector, filmstrip timeline with frozen-frame hold segments.

The shell borrows VSCode's layout: an activity bar, resizable rails, and a script editor where prose renders like a document (KaTeX inline) but every block is directly editable. Beside it sit a 16:9 program monitor with hover media controls and an NLE timeline with filmstrip clips, trim handles, drag-to-reorder, waveforms, and click-anywhere scrubbing on a wall-clock-interpolated playhead.

Three WebKit bugs cost more hours than anything else in the project:

The same debugging pattern cracked all three: build a harness that renders the real page in the real engine, then bisect with measurements instead of guesses.

TheoremExplainAgent (TIGER-AI-Lab, ACL 2025 oral) is the closest published system: a planner agent writes the story and narration, a coding agent writes Manim, and the output is a 5-minute-plus theorem video with audio. They also built TheoremExplainBench, 240 theorems across CS, chemistry, math, and physics at three difficulty levels, scored on five automated dimensions (accuracy, depth, logical flow, visual relevance, element layout). Best reported agent: o3-mini at a 93.8% render-success rate, 0.77 overall. Manimator (2025) maps research papers to Manim explanations.

Running TheoremExplainBench's prompts through this drafter and scoring on their five dimensions is a planned experiment.

Results

Demo clip pending. Studio chat: "create a lesson from my notes" → drafter trail → Build → play.

Takeaways

The single parser held everything together. Because only Python reads a lesson and everything else renders its JSON, the agents, the CLI, and the app never drifted apart over four days of constant change. There was simply nothing else to keep in sync.

Every agent failure that mattered (the wrong render quality, the blocked shell commands, the registry source-diving) got fixed by changing what a tool does by default. Adding more instructions never worked. And I only found the biggest failure, a pattern wasting about 20% of each run, by reading full transcripts; it was invisible from watching any single run live.

The last surprise was where the bugs lived. Three of the worst turned out to be WebKit's media and reactivity semantics rather than my own logic, and what caught all three was the headless harness driving the real page in the real engine.

Next steps

© 2026 Saeyohn Mahendran · home