Open-Source AI Pipeline Auto-Cuts Video Into Shorts

What happened

A developer who runs the anime YouTube Shorts channel @AnimeFactorio has open-sourced the core of his automated video-to-Shorts pipeline on GitHub (github.com/ialakey/shorts-factory). After three blog posts explaining the architecture, the face-tracking camera, and why raw transcript-plus-LLM editing produces garbage, the recurring request was the same: show the code. This release is a showcase branch of a larger private project — a trimmed MVP that keeps the full path from "long video in" to "finished vertical clip out" while cutting everything else.

The numbers are concrete: roughly 7,000 lines of core Python, 249 tests covering about 3,000 lines, and a fully annotated 344-line demo config file. The pipeline runs thirteen stages, each of which can be toggled independently, and there are no command-line flags — every channel's behavior is described in a single config.yaml, because the real work in a project like this is parameter tuning, and tuning needs a file you can diff and roll back.

Why it matters

Short-form video is now the primary discovery surface on YouTube, TikTok, and Instagram, and manually re-cutting long-form content into 9:16 clips does not scale past a handful of uploads per week. This pipeline automates the entire chain: episode.mp4 goes in, and the system produces a transcript, extracts signals from audio, faces, scene cuts, and pacing, runs an LLM to select the strongest moments, assembles segments, drives a virtual camera to reframe each shot to 9:16, layers in word-by-word subtitles, a watermark, and processed audio, strips and rebuilds metadata, and posts the result to Telegram.

What makes this more than another wrapper around FFmpeg and Whisper is the face-tracking camera logic, which cascades through MediaPipe, then YuNet (ONNX via OpenCV), then Haar Cascade, so a missing dependency never breaks the stage. Hero-subject selection uses a saliency score based on confidence, size, and centrality, with hysteresis parameters (face_switch_margin, face_switch_hold_s) that stop the camera from flip-flopping between speakers. Motion is smoothed with anti-jerk clamping and a low-pass filter, then driven by a physics model (acceleration proportional to position error minus velocity damping, with speed and acceleration limits and a dead zone) so the crop moves like an actual camera operator rather than a jittery auto-tracker.

How to use it today

The repository is structured so each of the three prior articles maps to a specific file. The architecture piece — independent modules instead of one end-to-end model, fail-soft behavior instead of fail-fast crashes, explicit intermediate artifacts — lives in core/channel_processor.py, where one channel is just a list of enabled stages. If a stage fails, the rest of the channel keeps running; if mediapipe is unavailable, face detection degrades instead of crashing; if an external service like Kodik is unreachable, that stage is skipped rather than halting the run. Every stage writes its output to disk next to the source clip: transcript.txt, moments.json, candidates.json, signals.json, and the LLM payload files, so you can inspect or reuse any intermediate result.

A debug: true flag pulls input from test_data/ and writes output to output_test_data/, dumping every intermediate value. If moment detection hasn't run yet but downstream stages (subtitles, watermark) are enabled, the debug mode falls back to processing the whole clip, so you can iterate on rendering without re-running Whisper and the LLM every time — a detail that matters because a pipeline that fails at stage nine and costs you the first eight stages of compute is a pipeline nobody will run iteratively.

Teams building their own short-form workflow don't need to replicate all thirteen stages from scratch. Prototyping the LLM moment-selection step or testing subtitle styling can be done faster with a general-purpose AI assistant first — [mykreatool.com](https://mykreatool.com) offers free AI tools for exactly that kind of quick iteration — before investing in a dedicated rendering pipeline like this one.

Who benefits

Anime and gaming channels repurposing long-form footage are the clearest fit, since that's what the pipeline was built for, but the same stage list — transcript, signal extraction, LLM moment selection, vertical reframing, subtitle burn-in — applies to podcasts, webinars, streams, and long-form YouTube content generally. Solo creators get the most leverage: instead of manually scrubbing a 20-minute episode for clip-worthy moments, the LLM ranks candidates and a human just approves or rejects. Small teams running multiple channels benefit from the per-channel config.yaml model, since tuning one channel's face-switch sensitivity or watermark position doesn't touch another's. Developers get a working reference for problems that are otherwise hard to find documented: cascading face detectors, hysteresis-based subject switching, and a fail-soft stage orchestrator.

Risks

This is a showcase branch, not the full production system — the author explicitly cut features to keep the MVP readable, so expect gaps compared to what actually runs the @AnimeFactorio channel. Running the full stack requires Whisper for transcription and an LLM API for moment selection, both of which cost money per episode and add external dependencies outside the developer's control. Face detection accuracy depends on video quality and content type; anime with stylized character proportions is not the same detection problem as live-action footage, and results on other content types are unverified. Auto-posting to Telegram and any downstream platform also means metadata stripping and re-encoding need to be checked against each platform's current policies, since automated repurposing of third-party video content carries copyright and platform-terms risk that the code itself does not resolve.

Conclusion

The shorts-factory release turns three years of blog-post theory into inspectable, testable code: a fail-soft, config-driven pipeline that takes long video to finished vertical clips in thirteen toggleable stages, backed by 249 tests. For creators and small teams drowning in re-cutting work, it's a concrete starting point rather than a black-box SaaS tool — read the config, run the demo, and adapt the stages that matter for your own content.