As many of you know, I'm always trying to share knowledge — whether through videos, talks, or interviews. And from every session, I like to extract the most powerful moments and turn them into Shorts.

But there’s a problem.

Most tools that can help me do this easily are paid. And they don't cost just a few bucks; they're quite expensive.

Now, I'm not being cheap. But I thought:

An engineer built this tool. I'm an engineer too. So I can build my own."

And then I reflected:

Well... not just me. **Me + Claude. \ And that's how Video Wizard was born — an open-source, AI-powered tool for turning long-form video into short-form content with subtitles, smart cropping, and professional templates.

In this post, I'll walk you through the architecture, the tech decisions, and what I learned building a production-grade video processing pipeline as a side project.

What Does Video Wizard Do?


Here's the full user flow:

  1. Upload a video (or paste a YouTube URL)
  2. AI transcribes the audio using OpenAI Whisper
  3. GPT analyzes the transcript and detects the most viral moments (scored 0-100)
  4. Smart cropping with face detection (MediaPipe + OpenCV) reframes for vertical
  5. Edit subtitles in a visual editor — merge, split, clean filler words
  6. Pick a template — 9 professional caption styles built as React components
  7. Render the final video with Remotion
  8. Download— ready for TikTok, Reels, or YouTube Shorts

But it's not just a clip extractor. There's also a standalone Subtitle Generator that skips the AI analysis and goes straight from upload to rendered subtitles, as well as a Content Intelligencetool for analyzing transcripts without video.

The Tool’s Architecture: Three Services, One Goal

I chose a microservices monorepo approach with Turborepo. Three independent services, each doing what it does best:

Why three services?

Key Insight: Video as React Components

This was the "aha" moment for me.
I choseRemotion because it lets you treat video like a UI. Components, props, composition — but applied to audiovisual content.

Smart Face Tracking: The Python Side

The most interesting algorithmic challenge was the smart cropping. When you convert a 16:9 video to 9:16, you need to decide where to crop — and ideally, you follow the speaker's face.


The processing engine uses MediaPipe's BlazeFace model for detection, then applies aweighted scoring algorithmwhen multiple faces appear


The result? Cinematic-quality camera movement that tracks the speaker without the "security camera" feel. And it's smart enough to skip face detection entirely when the source video already matches the target aspect ratio.

Subtitle Synchronization: The Hard Part Nobody Talks About

Getting subtitles to sync properly across three services with different time formats was the trickiest part.

Stage: Whisper output
Format: Seconds
Example: { start: 0.5, end: 2.3 }

Stage: Frontend editor
Format: Milliseconds
Example: { start: 500, end: 2300 }

Stage: Remotion renderer
Format: Seconds
Example: { start: 0.5, end: 2.3 }

One early bug had 60-second videos rendering as 0.06 seconds because the times were getting divided by 1000 twice. Fun times.

I also added a configurable 200ms subtitle offset to account for the perceptual delay between hearing a word and reading it:

const SUBTITLE_OFFSET = 0.2; // seconds
const adjustedTime = currentTime - SUBTITLE_OFFSET;

It’s a small detail, but it makes the subtitles feel perfectly synced.

The Subtitle Cleanup Toolkit

Beyond basic editing, I built automated detection for common subtitle issues:

These are pure functions — no side effects, easily testable:

const result = detectIssues(subtitles, config);
const cleaned = removeDetectedIssues(subtitles, result.issues);

With one click, subtitles go from raw Whisper output to clean, professional captions.

Architecture Decisions I’m Proud Of

Screaming Architecture

The folder structure tells you what the app does before you read a single line of code:

features/video/components/   // Renders video UI
hooks/                       // Manages video state
containers/                  // Orchestrates video workflows
types/                       // Defines video data shapes
lib/                         // Provides video utilities

Strict Separation of Concerns

API routes are thin. They only handle HTTP and delegate to services:

export async function POST(request: NextRequest) { 
  const body = await request.json(); 
  const data = await subtitleGenerationService.generateSubtitles(body); 
  return NextResponse.json({ success: true, data }); 
}

All business logic lives in service classes — reusable, testable, and independent of HTTP:

export class SubtitleGenerationService { 
  async generateSubtitles(input) { // 1. Call Python transcription // 2. Convert time formats // 3. Structure response }
  async renderWithSubtitles(input) { // 1. Send job to Remotion // 2. Poll until complete // 3. Return video URL } 
}

Zod Everywhere

Every external boundary is validated with Zod. Types are inferred, not duplicated:

const BrandKitSchema = z.object({ 
  logoUrl: z.string().optional(), 
  logoPosition: z.enum(['top-left', 'top-right', 'bottom-left', 'bottom-right']), 
  logoScale: z.number().min(0.1).max(2), 
  primaryColor: z.string().regex(/^#[0-9A-Fa-f]{6}$/).optional(), 
});

type BrandKit = z.infer<typeof BrandKitSchema>;

Types come from schemas — not the other way around.

The Tech Stack

What I Learned

  1. Engineer + AI = Multiplied Output

This project would have taken months working solo. With Claude as a pair programmer, I went from idea to working prototype significantly faster. Not because the AI wrote everything, but because it accelerated the tedious parts — boilerplate, FFmpeg flags, Remotion configuration — so I could focus on architecture and product decisions.

  1. Build Your Own Tools

We increasingly depend on external SaaS for everything. Sometimes the best way to learn is to build the tool yourself. You’ll understand video processing, ML pipelines, and rendering engines at a depth that no tutorial can give you.

  1. Time Formats Will Haunt You

If you’re building anything with subtitles, pick one time format (seconds or milliseconds) and stick with it. Document your conversions. Test edge cases. Your future self will thank you.

What’s Next?

This is still a personal project, but it already includes:

Roadmap ideas:

Try It Yourself

The repo is open source. You can run the entire stack locally

I’d love your feedback:

PRs, issues, and stars are all welcome.

GitHub: https://github.com/el-frontend/video-wizard

Built with Next.js, Python, Remotion, and a lot of help from Claude.