talkies: Tell OpenAI’s Speech API to Go Fuck Itself

Posted on by

I got the OpenAI Whisper API bill for a month of hooking my own tooling up to it and just sat there staring at the number like it had personally insulted my mother. Not because it was some insane amount — it wasn’t — but because I was paying per-minute rent on a model that’s been public and self-hostable for years, for the privilege of uploading my own audio to somebody else’s server so they could run inference I could run myself on a GPU I already own. Then I needed TTS on top of that for the same pipeline, which meant a second vendor, a second API key, a second bill, a second Terms of Service document nobody reads, and a second place where my data — voice samples, transcripts, whatever I fed it — sits on a stranger’s disk. I already had a bad taste in my mouth from building qwenspeak earlier — shoving text through Kokoro over SSH because I didn’t want a TTS API bill either. That thing worked, but it was SSH-shaped, single-purpose, and had zero interest in speaking OpenAI’s wire format. When aigate needed a speech backend that its OpenAI-compatible router could talk to without writing a custom adapter, none of my existing junk drawer fit. So I built talkies: one container, ASR in, TTS out, wired to speak the exact HTTP shape the entire OpenAI SDK ecosystem already understands, running on hardware I own.

Self-Hosting Speech Is a Dependency Swamp

Self-hosting speech models sounds simple until you actually try to assemble a service out of the pieces that exist in the wild. Here’s what you run into:

  • Every project has its own wire format. faster-whisper wants a script. NeMo wants you to fight its config system for an afternoon before it’ll even load a checkpoint. Kokoro’s a PyPI package you have to build an HTTP layer around yourself. None of them agree on a request/response shape, so if you’ve already got tooling built against OpenAI’s /v1/audio/transcriptions and /v1/audio/speech, you get to write a translation layer for every single one, then maintain it forever.
  • ASR-only or TTS-only. Pick a self-hosted ASR project and you’ll get exactly that — transcription, nothing else. Want TTS too? New container, new port, new config, new failure mode when one of them falls over and the other doesn’t.
  • GPU memory doesn’t negotiate. Load two heavy models on the same card without an eviction strategy and you get an OOM crash mid-request, or you provision a card twice the size you need so both models can sit resident forever doing nothing 95% of the time.
  • Voice cloning is either missing or bolted on badly. A lot of self-hosted TTS stacks either don’t do cloning at all, or they do it through some bespoke script you run once, offline, that spits out a checkpoint you then have to wire back into the serving path by hand.
  • CPU vs GPU is an afterthought. Most projects assume you have a GPU sitting idle, or they run everything on CPU and eat five minutes per transcription. Nobody ships both and tells you honestly which models make sense on which.

None of this is exotic. It’s the standard self-hosted-AI tax: every component speaks its own dialect, and gluing them together into something an existing client library can actually use is the real work nobody writes about.

One Wire Shape, Every Backend

talkies is a FastAPI service (psyb0t/docker-talkies) that speaks OpenAI’s speech wire format on both directions — POST /v1/audio/transcriptions for ASR, POST /v1/audio/speech for TTS — and dispatches every request to one of a handful of backend engines based on a model slug you pass in the request, exactly the way you’d pick a model name against the real OpenAI API. Point the official openai SDK at it, change the base_url, done:

from openai import OpenAI
c = OpenAI(base_url="https://ciprian.51k.eu00/v1", api_key="x")
c.audio.transcriptions.create(model="whisper-large-v3-turbo", file=open("a.mp3", "rb"))
c.audio.speech.create(model="qwen3-tts-0.6b", voice="alloy", input="hello").stream_to_file("out.mp3")

Behind that identical wire shape, the model registry (models.json, or models-cpu.json for the CPU image) maps every slug to an executor string, and talkies/config.py validates that string against a fixed allowlist — VALID_EXECUTORS — of exactly eight values: whisper, parakeet, parakeet_cpp, canary_multitask, canary_salm, kokoro, kokoro_nvidia, qwen3_tts. Anything outside that set fails the container at boot instead of shipping a half-broken service. The factory in talkies/models/__init__.py reads the registry and instantiates the matching backend class per slug — each one implements a duck-typed protocol (get_model(), unload(), loaded(), last_used_secs_ago(), plus transcribe() for ASR or synthesize()/voices() for TTS), so the route handlers in server.py never care which engine is actually doing the work.

The shipped models.json registers seven ASR slugs (two Whisper sizes via faster-whisper, Parakeet-TDT, Nemotron-3.5-ASR via a C++ ggml runtime, and three Canary variants) and seven TTS slugs across three engine families (Kokoro in two runtime flavors, plus five Qwen3-TTS checkpoint/mode combinations). Whatever the README’s badges or bullet lists say about the count — and they disagree with each other in three different places, which is exactly the kind of drift you get when a project grows fast — the registry file is the actual contract, and I counted it directly rather than trust prose. Fourteen total slugs on the CUDA image; on the CPU image, models-cpu.json drops the models that are useless without VRAM and keeps four ASR slugs (both Whisper sizes, Canary-180M-Flash, and Nemotron-3.5-ASR — the last one runs fine on CPU because it’s a ggml C++ port, not a PyTorch model) plus two Kokoro TTS variants.

Quick Start

docker run -d --name talkies 
  -v $HOME/talkies-data:/data 
  -p 8000:8000 
  psyb0t/talkies:latest
curl -s https://ciprian.51k.eu00/v1/audio/transcriptions 
  -F "file=@samples/hello.wav" 
  -F "model=whisper-large-v3-turbo" | jq

First boot downloads every enabled model into /data/models/<slug>/ as a flat directory — no HuggingFace cache indirection, just snapshot_download(local_dir=...) straight to a path keyed by the slug. Bind-mount /data or you re-download gigabytes every restart. If you don’t want the whole registry sitting on disk, TALKIES_ENABLED_MODELS whitelists slugs — set it to a comma-separated list and the prefetch loop, plus /v1/models, only serves what’s in it. Reference an unknown slug and the container fails fast at startup with the full catalog printed so you can fix the typo instead of debugging a 404 an hour later:

docker run -d --gpus all 
  -e TALKIES_ENABLED_MODELS=whisper-large-v3-turbo,qwen3-tts-1.7b-custom 
  -v $HOME/talkies-data:/data 
  -p 8000:8000 psyb0t/talkies:latest-cuda

ASR: Seven Slugs, One Whisper-Shaped Response

Every ASR backend, regardless of what’s actually crunching the audio underneath, returns the same Whisper-shape response — text, and for verbose_json, full segments and words arrays. Swap model=whisper-large-v3 for model=canary-1b-flash and nothing downstream of the HTTP response has to know or care.

faster-whisper (2 slugs)

whisper-large-v3 and whisper-large-v3-turbo run through faster-whisper‘s CTranslate2 runtime — both CPU- and GPU-capable, both in the CPU image.

NeMo (4 slugs)

parakeet-tdt-0.6b-v3 (TDT decoder), canary-180m-flash and canary-1b-flash (multitask transformer heads, the 1B variant does EN/DE/FR/ES speech-to-text translation both directions), and canary-qwen-2.5b, which swaps Canary’s decoder for a Qwen2 LLM — NVIDIA’s “speech-augmented language model” trick. Canary-Qwen has no alignment head, so it’s the one backend that comes back with empty segments/words arrays in verbose_json — you still get the full transcript, you just don’t get per-word timestamps for that one model.

parakeet.cpp (1 slug)

nemotron-3.5-asr-0.6b is the odd one out — a GGUF quant of NVIDIA’s Nemotron-3.5-ASR-Streaming-0.6B served through mudler/parakeet.cpp, a C++17/ggml runtime, loaded via ctypes from talkies/models/parakeet_cpp.py with no Python NeMo anywhere on the hot path. It’s the one autoregressive-streaming-class ASR model that runs decently on plain CPU, which is why it ships in both images while its siblings (Parakeet-TDT, Canary-1B, Canary-Qwen) are CUDA-only. twenty-three pinnable locales plus auto detect straight out of the registry entry’s languages array, per-word timestamps synthesized into segments by grouping on silence gaps (_SEGMENT_GAP_THRESHOLD_S = 0.5 in parakeet_cpp.py).

Long files get sliced first — anything over TALKIES_VAD_CHUNK_THRESHOLD (default 30 seconds) goes through Silero VAD, split into speech regions capped at TALKIES_VAD_MAX_SPEECH (default 28 seconds), transcribed per-chunk, and stitched back into one continuous timeline with offset-corrected timestamps. Every backend goes through the same chunker — Whisper’s own internal long-form window gets bypassed entirely so the chunking behavior is identical across engines instead of Whisper doing its own thing while NeMo does another.

There’s also stereo diarization without a separate speaker-embedding model bolted on: feed a 2-channel file with diarization=true, left channel becomes speaker L, right becomes speaker R, each transcribed independently and merged chronologically. Mono input with diarization=true gets rejected with a 400 (NotStereoError, checked via ffprobe channel count in talkies/audio.py) — it’s not magic speaker separation, it’s a two-mic-setup trick, and it says so plainly rather than pretending otherwise.

TTS: Kokoro Twice, Qwen3 Five Ways

Three TTS engine families, seven slugs. kokoro-82m runs the 82M-parameter open-weight Kokoro model in-process via the kokoro PyPI package — fast enough on CPU to be genuinely useful, no sidecar. kokoro-82m-nvidia is the same weights served through ONNXRuntime against NVIDIA’s TensorRT-friendly ONNX export instead — CUDA execution provider on the GPU image, CPU EP on the CPU image, G2P via espeak-ng/phonemizer instead of misaki. Same voice catalog, same wire format, drop-in swap between the two.

Voices are scanned live off disk — talkies/models/kokoro.py filters the voice pack down to fourteen name prefixes across six languages (American and British English, Spanish, French, Hindi, Italian, Portuguese) that run on the lightweight espeak-ng G2P shipped in the base image, skipping the Japanese and Mandarin voices that need heavier misaki extras nobody asked for. Kokoro has no OpenAI voice aliases — you get the native af_heart / bm_george / ef_dora-style names, discoverable via GET /v1/audio/voices.

The other five TTS slugs are all Qwen3-TTS, CUDA-only because the underlying faster-qwen3-tts wrapper captures CUDA graphs at load time and has no CPU code path at all:

  • qwen3-tts-0.6b / qwen3-tts-1.7b — base mode, voice cloning. Drop a 10-30 second reference .wav into /data/custom-voices/<name>.wav, synth as voice=<name>. Nested paths survive (clients/acme/jane.wav becomes voice clients/acme/jane). Add a sibling <name>.txt transcript and cloning quality jumps noticeably (in-context-learning mode); skip it and the backend falls back to x-vector-only synthesis with a logged warning instead of failing outright.
  • qwen3-tts-0.6b-custom / qwen3-tts-1.7b-custom — nine fixed preset speakers, no reference audio needed. The 1.7B variant honours an instructions field as an emotion cue (“Speak angrily.”); the 0.6B checkpoint doesn’t support it at all — faster-qwen3-tts nullifies the field internally, and talkies logs a warning rather than silently eating your instruction.
  • qwen3-tts-1.7b-design — no voice catalog at all. You describe a voice in natural language via instructions (“a warm, friendly young female voice with a cheerful tone”) and the model invents one. Empty instructions is a 400, and re-running the same description twice won’t give you the same voice back — sampling is stochastic, that’s the deal.

Per-request sampling controls (temperature, top_k, top_p, repetition_penalty, max_new_tokens, do_sample) ride along as OpenAI-extra fields via extra_body on the official SDKs — none of this needed a custom endpoint, it’s all still POST /v1/audio/speech. Output gets encoded through ffmpeg into whichever of mp3 / opus / aac / flac / wav / pcm you asked for — that’s the exhaustive list straight out of talkies/tts.py‘s format table, not a guess. pcm against a Qwen3-TTS model streams raw chunked bytes instead of buffering the whole utterance, which is the only response format that actually streams — everything else, Kokoro included, synthesizes the full clip before returning it.

Resource Management, File Staging, and MCP

All backends share the same VRAM/RAM pool, one model resident at a time. When a request comes in for a slug that isn’t currently loaded, whatever else is loaded gets evicted first — sibling eviction, regardless of modality, so loading Kokoro kicks out a resident Whisper and vice versa. There’s also an idle sweeper on a TALKIES_SWEEPER_INTERVAL cadence (default 60s) that unloads anything idle past TALKIES_MODEL_TTL (default 600s / 10 minutes; set it to 0 to disable auto-unload). The Ollama-style introspection surface — GET /api/ps, DELETE /api/ps/{model_id}, POST /unload — lets you check what’s resident and evict it manually, and the whole thing mirrors speaches‘s resource-management shape closely enough that the same LiteLLM-style proxy driver code works against both.

Server-side file staging (/v1/files) exists so you’re not re-uploading the same audio bytes on every retry while you tune a response_format. PUT a file once, reference it by relative path afterward via the file_path form field on /v1/audio/transcriptions instead of the multipart file field. file_path also accepts a plain http(s):// URL — first hit downloads and caches it under a sha256-keyed path, every subsequent call with the same URL is a cache hit, and concurrent requests for the same URL don’t double-fetch. Path traversal (.., backslashes, null bytes, double slashes) gets rejected with 400, and symlinks pointing outside the staging root are refused after path resolution.

There’s also a full MCP server mounted at /v1/mcp over Streamable HTTP, running in the same FastAPI process and sharing the exact same backend pool and auth middleware as the HTTP routes — a model an MCP tool call loads is the same instance the HTTP API sees. Six tools total, straight out of mcp_server.py: list_models, transcribe, list_files, put_file, get_file, delete_file. Wire it into Claude Code in one line:

claude mcp add --transport http talkies https://ciprian.51k.eu00/v1/mcp

Which means an agent can transcribe a recording or shove audio around the staging dir as tool calls without you writing any glue — same server, same models, same eviction rules, just a different transport. TTS deliberately isn’t on the MCP surface: list_models drops anything that doesn’t implement transcribe(), so synthesis stays on POST /v1/audio/speech where the streaming and format machinery already lives.

Auth, and Not Trusting the Network by Default

Set TALKIES_AUTH_TOKEN and every route except /healthz and CORS preflights requires Authorization: Bearer <token> or gets a 401 with WWW-Authenticate: Bearer attached. It’s implemented as ASGI middleware — not a FastAPI dependency — specifically so it covers the mounted MCP sub-app too, and the token comparison runs through hmac.compare_digest instead of a plain string equality, so there’s no timing side-channel to lean on. Leave the var unset and the server’s wide open, which is the deliberate self-hosted-LAN default — stick a reverse proxy in front if that’s not your threat model.

Everything else is boringly locked down the way a production image should be: base images pinned by sha256 digest, every Python dependency hash-locked via uv.lock and --require-hashes installs for the heavy ML stack, an exclude-newer gate in pyproject.toml that refuses to lock package versions newer than the day the lockfile was generated (kills same-day supply-chain nonsense before it can land), runs as non-root, and HF_HUB_OFFLINE=1 in steady state — once the weights are cached, the container has zero reason to ever hit the network again except to serve your requests. URL downloads via file_path get an optional SSRF guard (TALKIES_BLOCK_PRIVATE_DOWNLOADS) that refuses hostnames resolving to private/loopback/link-local ranges, off by default because most self-hosted deployments are LAN boxes fetching from other LAN boxes, but there if you’re exposing this to something less trusted.

What It Replaced

talkies is the thing I wished existed before I built qwenspeak, before aigate needed a speech backend, before I got tired of two separate vendor bills for two halves of the same feature. One container, one wire format, seven ASR slugs, seven TTS slugs, voice cloning, an MCP endpoint, and none of it phones home once the models are cached on disk. If you’re already speaking OpenAI’s SDK against a real API key, pointing it at this instead is a base-URL change, not a rewrite.

Repo’s at github.com/psyb0t/docker-talkies, image’s on Docker Hub. WTFPL-licensed, so do whatever the fuck you want with it.