flickies: A Video Toolbox That Happens to Do Lipsync

Posted on by

I was neck-deep in an avatar pipeline and needed two unglamorous things at once: a lipsync pass, and a fistful of ffmpeg operations around it — trim the source, mux the audio, transcode the output, cut a thumbnail sheet. Every “solution” I found was some SaaS wrapper around a model I couldn’t inspect, billed per second of output, gated behind an API key, with a watermark baked into the corner because god forbid I pay $40/month and still own my own render. One of them wanted my face video AND my audio uploaded to their servers before it would even quote me a price. Another had a “commercial license” tier that cost more than my GPU. I sat there thinking: I have a 3060 sitting in a box doing nothing after 6pm, ffmpeg has existed since God was a boy, and Wav2Lip has been open source since 2020. Why the hell am I renting this from a stranger.

So I didn’t. flickies is the video half of the self-hosted toolkit I’ve been building out — docker run, point it at a face and an audio track, get an mp4 back. No account, no per-second meter, no watermark, no uploading my shit to someone else’s inference farm. It’s the sibling of audiolla (audio) and talkies (speech) — same async-job model, same bind-mount-/data story, same opt-in non-commercial gate, same “one port, zero cloud” attitude. It slots straight into aigate as the video engine behind the same nginx front door as everything else I run.

Renting Your Own Face Back

Cloud lipsync-as-a-service is a special kind of racket, and I’ve got a list:

  • You’re uploading a human face to a stranger’s GPU. Not a cat picture. A face, doing whatever the fuck you told the audio to make it say. That data doesn’t evaporate after the render finishes — it sits on someone else’s disk with someone else’s retention policy.
  • Billing per second of output turns experimentation into an accounting exercise. Want to iterate on ten takes to get the sync right? Congrats, you just paid ten times.
  • Watermarks and tier walls — the “free” tier slaps a logo on your output, the paid tier that removes it costs more than the hardware that would run this locally in a weekend.
  • Nobody tells you the license story. A shocking number of these SaaS wrappers are sitting on top of Wav2Lip, which is trained on LRS2 — a dataset with an explicit non-commercial clause. The SaaS charges you money to run a non-commercial model and just… doesn’t mention that part. That’s not my problem to solve for you if you self-host, but at least I make you actively flip a switch to acknowledge it instead of hiding it in a ToS nobody reads.
  • Zero introspection. You get a black box REST endpoint and a “trust us” — no idea which model variant ran, what settings, whether your face restore pass is chained on or off.

And on the ffmpeg side of things — trimming, transcoding, slapping audio onto a video, pulling a thumbnail grid — every “video processing API” I found for that was somehow ALSO a paid product, for operations that are one ffmpeg invocation with the right flags. I already solved plain file-processing-over-a-wire with mediaproc; flickies does the same “just shell out to a real tool, stop reinventing it” thing but scoped specifically to video and wired for the ML half too.

Spec-First and Container-Shaped

flickies is a FastAPI service that speaks one wire format for two very different kinds of work: pure ffmpeg CPU operations, and GPU model inference for lipsync and face restore. Every video-producing endpoint takes the same shape of request: exactly one input (file_path staged locally, or file_url the server fetches for you), and exactly one output (output_path written under FILES_DIR, or output_url the server PUTs the result to — presigned S3 URL, whatever). Mix and match freely. Stage a file locally, get a presigned URL back. Fetch from a URL, write to local disk. It doesn’t care.

The ML side runs through a registry that manages one GPU pool with hot-swap eviction: request wav2lip, it loads. Request gfpgan next, the registry evicts wav2lip first (del the refs, gc.collect(), then torch.cuda.empty_cache() — in that exact order, because PyTorch model graphs hold reference cycles and skipping the gc.collect() step means “unloading” a model doesn’t actually free the VRAM, which is a bug I shipped in v0.1.0/v0.2.0 and fixed for real in v0.3.1). A background sweeper also unloads whatever’s resident once it’s been idle longer than FLICKIES_IDLE_UNLOAD_SECS (default 600s). One model lives in VRAM at a time; that’s the whole design.

Everything downstream of that — the routes, the request/response shapes, the error codes — comes from one file: openapi.yaml. It’s not documentation-after-the-fact, it’s the actual generator input for three separate things: the server’s own Pydantic validation models, the Go client, and the Python client. Change the spec, run make generate, all three regenerate together. make generate-check is a CI gate that fails the build if any of them drift from the spec. I’ll get into why that matters below because it’s the part of this project I’m most smug about.

Quick Start

docker run -d --name flickies 
  -v $HOME/flickies-data:/data 
  -p 8000:8000 
  psyb0t/flickies:latest
curl -s -X POST https://ciprian.51k.eu00/v1/video/info 
  -H "Content-Type: application/json" 
  -d '{"file_path": "uploads/clip.mp4"}' | jq

Two images: psyb0t/flickies:latest (CPU, python:3.12-slim base) and psyb0t/flickies:latest-cuda (nvidia/cuda 12.4 runtime base). The CPU image runs every ffmpeg operation plus Wav2Lip on CPU — slow, but real; the CUDA image runs everything at a speed you’d actually tolerate. Tested target is an RTX 3060 12GB.

Four ML Engines: Lipsync and Face Restore

engines.json defines exactly four ML engines, each with a slug, a CUDA requirement flag, a VRAM floor, and — where it matters — a license gate:

wav2lip / wav2lip-gan

Rudrabha/Wav2Lip, vendored, 96×96 native resolution. Two variants sharing one engine class, switched by a variant field: base (max sync accuracy, softer mouth) and gan (GAN refiner, sharper mouth, very slightly worse sync). Both device-select automatically — FLICKIES_DEVICE=auto checks torch.cuda.is_available() and falls back to CPU cleanly. On the initial release benchmark that’s ~44 seconds for a 3-second clip on CPU, ~22 seconds on GPU. That CPU number isn’t a joke — it’s genuinely usable for short clips, which is more than I can say for most “GPU required” open source lipsync repos that just crash on a CPU box instead of degrading gracefully.

latentsync-1.5

ByteDance’s LatentSync 1.5, Apache-2.0, pinned specifically to the 1.5 checkpoint because 1.6 wants 18GB of VRAM and my hardware ceiling is 12. SD-1.5 latent-space backbone, Whisper-tiny audio embeddings cross-attending into a UNet3D via AnimateDiff — heavier machinery than Wav2Lip, and it shows: ~170 seconds for a 6-second clip on the 3060, peaking around 9.6GB VRAM. This is the one engine in the whole set that hard-requires CUDA — the code checks torch.cuda.is_available() at load time and raises a 400 if it’s not there, no CPU fallback attempted. It’s also the default engine when the non-commercial gate isn’t opted into, because unlike Wav2Lip it carries no LRS2 baggage.

gfpgan

TencentARC’s GFPGAN v1.4, Apache-2.0. This chains after Wav2Lip to fix the soft, low-res mouth crop that 96×96 native inference leaves behind — Wav2Lip nails the sync, GFPGAN cleans up the visual mess around it. It also works standalone via POST /v1/video/restore if you just want a face-restore pass on existing footage. Same auto device-selection as Wav2Lip, falls back to CPU. Frame-by-frame: read via cv2, run the restorer per frame, write to a silent mp4, then mux the original audio back over the restored frames.

Weights for all four live in the standard HuggingFace cache layout under /data/hf/hub/models--<org>--<name>/ — content-addressed blobs, snapshot symlinks, reusable by anything else HF-aware sharing that bind mount. Lazy by default: each engine fetches its repo on first request. FLICKIES_PREFETCH_ALL=1 or a scoped FLICKIES_ENABLED_ENGINES=wav2lip,gfpgan pulls weights at boot before uvicorn even starts, so your first real request doesn’t eat a multi-minute cold-download penalty.

The license gate isn’t decorative

Wav2Lip’s weights are trained on LRS2, a non-commercial dataset. flickies doesn’t bury that in a README nobody reads — the server code physically refuses to load either wav2lip variant unless FLICKIES_ENABLE_NONCOMMERCIAL=1 is set in the server environment. Try to acquire the engine without it and require_noncommercial_optin() raises a NonCommercialOptInRequired, which the API surfaces as a proper error instead of a silent 500. LatentSync 1.5 and GFPGAN are both Apache-2.0, no gate, load freely. Same mechanism audiolla uses for its own MusicGen / matchering non-commercial gates — I stole the pattern from myself, which I’m allowed to do.

Seven ffmpeg Operations, Because Not Everything Needs A GPU

Half of what people actually need from a “video API” isn’t ML at all — it’s ffmpeg with sane defaults and error handling. flickies exposes seven pure-ffmpeg operations, no model loaded, pure CPU, available in both images:

  • trim — cut to [start_sec, end_sec]. Default mode is -c copy stream-copy (fast, but snaps to the nearest keyframe — can eat up to a GOP of leading content). Set precise: true and it re-encodes via libx264 -crf 18 -preset veryfast + AAC 192k for frame-accurate boundaries instead.
  • concat — join 2+ videos in order via the concat demuxer. Same stream-copy-vs-precise tradeoff as trim; precise: true re-encodes through the demuxer with uniform codec params so mismatched-encoder inputs actually join instead of corrupting.
  • transcode — universal re-encode across mp4/webm/mov/mkv, with codec/crf/preset/fps overrides. Also handles gif output as a special two-pass path: palettegen then paletteuse through a filter_complex, because a naive ffmpeg-to-gif conversion looks like garbage and everyone knows it.
  • scale — resize to width×height, optional aspect-ratio-preserving pad.
  • mux_audio — replace or merge an audio track into a video.
  • extract_audio — pull the audio track out as wav/mp3/m4a/ogg/flac.
  • thumbnail_grid — sprite-sheet PNG via the thumbnail + tile filters, rows/cols/cell-size all configurable.

There’s an eighth video capability that isn’t in that “ops” list because it doesn’t produce a video — /v1/video/info shells to ffprobe and hands back duration, codec, fps, dimensions, bitrate. Every single one of these runs through one choke point in ffmpeg.py that shells out via asyncio.create_subprocess_exec, captures stderr, and raises a structured FFMPEG_FAILED error on non-zero exit instead of leaking a raw traceback.

Async Jobs And Webhooks, For When You’re Not Going To Sit There Waiting

LatentSync at 170+ seconds a clip is not something you want to hold an HTTP connection open for. Every video-producing endpoint accepts async_job: true (or you just omit both output fields and it’s implied): the server pre-allocates a job id, schedules the work as a background asyncio task, and returns 202 {job_id, status: "accepted"} immediately. Poll GET /v1/jobs/{job_id} for pending → running → complete/failed/cancelled. If you don’t want to poll, pass a webhook_url and the server delivers the final job state itself.

The webhook delivery isn’t a fire-and-forget POST and a shrug. It’s HMAC-SHA256 signed over timestamp + "." + body, sent as X-Webhook-Timestamp + X-Webhook-Signature: t={ts},v1={hex} headers, with a real exponential backoff retry schedule on anything that isn’t a 2xx: 30s, 1m, 5m, 30m, 2h, 12h, then it logs a dead-letter entry and gives up. Your receiver is expected to de-dupe on (timestamp, signature) for idempotency. This is the same shape as a payment provider’s webhook contract, because that’s the only prior art worth copying for “reliably tell someone else a long job finished.”

Eleven Tools On The MCP Wire

Mounted at /v1/mcp as JSON-RPC over streamable HTTP, flickies exposes eleven MCP tools that mirror the REST surface almost 1:1: list_engines, info, lipsync, restore, transcode, trim, concat, scale, mux_audio, extract_audio, thumbnail_grid. Point a function-calling LLM at it — LibreChat, Cursor, Claude with the MCP connector, whatever agent framework you’re running — and it can drive the entire pipeline itself: stage a face video, stage an audio clip, call lipsync with restore_face: true to chain GFPGAN automatically after the sync pass (which, worth noting, triggers a hot-swap eviction of the lipsync model mid-tool-call — that’s intentional, it frees the VRAM the restore pass needs), then hand back a path or a size.

This is also the layer aigate proxies to when you toggle FLICKIES=1 / FLICKIES_CUDA=1 — a single nginx front door in front of every one of my self-hosted AI services, this one included.

Spec-First: Generated Go And Python Clients From The Same Damn File

This is the part that actually separates flickies from “yet another ML wrapper API” for me. openapi.yaml is not decoration written after the code to look professional — it’s the single source of truth that three separate artifacts are generated FROM, not written to match:

make generate              # regenerate all three: server models + Go client + Python client
make generate-models       # just server-side Pydantic (src/flickies/schema/_generated.py)
make generate-client-go    # just the Go client (pkg/clients/go/client.gen.go)
make generate-client-python # just the Python client (pkg/clients/python/flickies-client/)
make generate-check        # CI gate — fails the build if generated files drift from openapi.yaml

Never hand-edit generated files. Edit the spec, run make generate, commit everything together. The Go client comes from oapi-codegen, the Python client from openapi-python-client — both real, typed, importable packages, not curl-wrapped-in-a-function afterthoughts:

go get github.com/psyb0t/docker-flickies/pkg/clients/go@latest
import flickies "github.com/psyb0t/docker-flickies/pkg/clients/go"
c, _ := flickies.NewClient("https://ciprian.51k.eu00")
resp, err := c.PostVideoLipsync(ctx, flickies.VideoLipsyncRequest{...})
pip install "git+https://github.com/psyb0t/docker-flickies.git#subdirectory=pkg/clients/python/flickies-client"
from flickies_client import Client
from flickies_client.api.lipsync import post_video_lipsync
from flickies_client.models import VideoLipsyncRequest
client = Client(base_url="https://ciprian.51k.eu00")
result = post_video_lipsync.sync(client=client, body=VideoLipsyncRequest(...))

It’s not perfect — the Go client’s VideoTrimRequest / VideoConcatRequest structs currently drop start_sec/end_sec/precise because of a known oapi-codegen limitation with allOf-composed schemas (the inline properties block on the composed type gets dropped). Go callers marshal those specific bodies via map[string]any in the meantime. I’d rather document a real generator limitation than pretend the pipeline is flawless — the point isn’t that codegen is magic, it’s that the spec and every client that speaks it are mechanically incapable of drifting apart, because they all come from the same build step.

Logging, Auth, Rate Limits — The Boring Shit That Actually Matters At 3AM

Structured JSON logging goes to stderr AND a rotating file (FLICKIES_LOG_FILE, default 50MB × 5 backups) with a custom formatter that recursively redacts anything matching password|token|secret|api_key|authorization|cookie|hf_*|sk-ant-* in both keys and values, at format time, before the line is ever written. Every log record carries a trace_id and request_id threaded through a ContextVar, seeded from the inbound X-Request-Id header if it’s shape-valid (UUID v4 or ULID, ≤64 chars, no newlines — garbage input just gets a fresh UUID minted instead of being echoed back, which closes a log-injection vector). Set FLICKIES_LOG_LEVEL=DEBUG and you get reconstruction-grade tracing: every ffmpeg/ffprobe command, every trim/concat decision between stream-copy and precise re-encode, engine inference wall-clock time, URL fetch/upload byte counts, job lifecycle transitions. Logged URLs are query-string-stripped first, so a presigned output_url doesn’t leak its signature into your log files.

Auth is a static bearer token via FLICKIES_AUTH_TOKEN, checked with hmac.compare_digest so it’s not timing-attackable, /healthz exempt so your orchestrator’s probes keep working. Unset the env var and auth is just off — your call, your network boundary. On top of that: a per-IP token-bucket rate limiter (default 60 req/min, tunable), and an Idempotency-Key-based dedupe layer that caches (status, body) per (key, method, path) so a retried POST doesn’t double-run an expensive render. All three are pure-stdlib ASGI middleware — no extra dependency just to rate-limit requests.

Where It Lives

flickies mounts inside aigate at /flickies/ and /flickies-cuda/ behind the same nginx that fronts every other self-hosted AI service I run — one make run-bg, both variants toggled with FLICKIES=1 / FLICKIES_CUDA=1. It’s the video-shaped piece of the same puzzle audiolla and talkies fill for audio and speech — same wire contract, same operator ergonomics, so adding a fourth or fifth service to that stack later doesn’t mean relearning anything.

What You Actually Get

flickies is a video toolbox that happens to include lipsync from the start: four real engines behind one hot-swap GPU pool, seven ffmpeg operations that don’t pretend to be anything fancier than ffmpeg, async jobs with actually-signed webhooks instead of a fire-and-forget POST, eleven MCP tools for whatever agent you’re pointing at it, and typed clients in Go and Python generated from the exact spec the server validates against — not hand-maintained, not drifting, not lying to you about what the API actually accepts. Self-hosted, WTFPL, runs on a GPU you already own.

Grab it at github.com/psyb0t/docker-flickies or pull the images straight off Docker Hub. Do what you want with it — but stop paying a SaaS to run an open source model on hardware you could’ve bought outright with three months of their invoice.