audiolla: Thirty-Five Audio Engines Behind One Docker Port

Posted on by

A while back I needed a box that could rip vocals out of a track, tell me the BPM and key, transcribe a melody to MIDI, and clean up a shitty voice recording. That’s four jobs. That’s also four completely separate Python worlds: Demucs wants one torch version, basic-pitch wants its own ONNX runtime, pyannote wants a HuggingFace token and a transformers pin that fights with everything else in the room, and DeepFilterNet wants whatever the hell DeepFilterNet wants that week. I tried to put all of that in one venv like an idiot. Pip spent twenty minutes resolving a dependency graph before giving up and telling me to go fuck myself in dependency-conflict language. I tried separate venvs per tool, which technically works and also means I now have five different Python interpreters on one box, five different sets of CUDA-compatible torch wheels eating disk, and a shell script gluing subprocess calls together because none of these libraries were ever meant to talk to each other.

This is the normal state of self-hosted audio tooling. Every serious audio model — stem separation, mastering, MIR analysis, speech enhancement, diarization, audio-to-MIDI — ships as a standalone Python library with its own opinions about numpy, its own opinions about torch, and zero opinions about how you’re supposed to combine six of them in one place without your machine catching fire. I got tired of maintaining a Frankenstein of venvs and glue scripts every time I wanted to do more than one thing to a piece of audio, so I built audiolla: one Docker container, one HTTP port, every audio tool I actually use wired behind a single JSON API.

Every Tool, Its Own Dependency Hell

Here’s the actual list of grievances, not the polite version:

  • Every model is its own dependency hell. Demucs wants torch. matchering wants a specific numpy. pedalboard is a C++ extension that occasionally decides your platform doesn’t exist. pyannote wants transformers pinned to a version that breaks basic-pitch’s onnxruntime. None of this is anyone’s fault individually — it’s just what happens when a dozen research labs ship a dozen libraries with zero coordination.
  • CPU vs GPU is a build-time decision, not a runtime one. torch-cpu and torch-cuda are different wheels. You don’t get to decide at 2am that you actually want GPU inference this time — you rebuild the whole environment.
  • Licensing is a minefield. matchering and pedalboard are GPL v3. MusicGen weights are CC-BY-NC. Riffusion is CreativeML-OpenRAIL-M. Stable Audio Open has a community license with a revenue threshold. If you’re gluing these together yourself, you’re also the one who has to remember which model you’re legally allowed to point at a paying customer.
  • Every tool has its own calling convention. One wants a file path, one wants raw bytes on stdin, one wants a numpy array, one insists on writing its own output file next to the input because some researcher never anticipated anyone running this in a pipeline.
  • None of it talks to an LLM agent out of the box. If you want Claude or any other agent to actually drive audio processing as a tool call, you’re writing that adapter yourself, for every one of these libraries, forever.

I spent three days once just getting Demucs and pyannote to coexist in the same container without one of them silently downgrading the other’s torch. That’s three days I will never get back, spent doing pip archaeology instead of actual audio work. audiolla exists so nobody else has to burn that time.

One Service, Lazy-Loaded Engines

audiolla is a FastAPI service that wraps every one of these libraries behind a flat REST surface. The core idea: every engine is a config entry, not a hardcoded branch. There’s an engines.json registry — I counted it straight out of the file, no guessing — that maps a slug like htdemucs or uvr-denoise to an executor module, plus whatever metadata that executor needs (model file, stem list, CUDA requirement). The server reads that registry at startup, and GET /v1/engines hands the whole thing back to you as JSON so you always know exactly what’s loaded and what isn’t.

Engines are lazy-loaded and evicted. Nothing sits in VRAM or RAM until you actually hit it — the first request to htdemucs pulls the model, subsequent requests reuse the warm instance, and an idle sweeper (default TTL 600 seconds, tunable via AUDIOLLA_ENGINE_TTL) evicts anything that’s been sitting unused so a GPU box doesn’t slowly fill up with every model you’ve ever touched. GET /v1/ps shows you what’s currently resident, DELETE /v1/ps/{engine} evicts one by hand, POST /v1/unload nukes everything. It’s basically docker ps for neural nets running inside your neural net container, which is a sentence I never expected to type unironically.

There are two images: a CPU build (psyb0t/audiolla:latest) and a CUDA build (psyb0t/audiolla:latest-cuda), built off NVIDIA’s CUDA 12.6 runtime. The CPU image ships 27 of the 35 engines — missing three of the four Demucs variants (htdemucs_ft, htdemucs_6s, mdx_extra) and all five text-to-audio generators. The five generators plus htdemucs_ft carry an explicit cuda_only: true flag that the server enforces at load time — try to hit one on a CPU box and it raises instead of grinding away for an hour. htdemucs_6s and mdx_extra don’t even carry the flag, they’re just not listed in engines-cpu.json at all, which amounts to the same “not on this build” outcome but isn’t the same mechanism, and running a diffusion or transformer text-to-audio model on CPU would be a war crime against your own patience anyway. The CUDA image ships all 35. Same codebase, same API, the registry file is just swapped (engines.json vs engines-cpu.json) at build time.

The v1.0 API contract — JSON in, files staged, nothing raw

The API went through a breaking v1.0.0 rewrite and it’s the right kind of breaking. Before, every endpoint was multipart form data in, raw audio bytes out — fine for curl, miserable for anything that wants to chain calls or hand this off to an LLM agent. As of v1.0.0 the contract is spec-first: openapi.yaml is the actual source of truth and the Pydantic request models get generated from it via make generate, so the schema can’t silently drift from what the handlers accept. I went and read the route code directly instead of trusting the README on this, because the whole point of this rewrite is that the old behavior is dead and I didn’t want to publish stale examples.

The actual shape, confirmed in src/audiolla/server.py:

  • Multipart upload is gone everywhere except one route. PUT /v1/files/{path} is the only endpoint that takes a raw binary body — body = await request.body(), straight into files_mod.write_atomic(). Every other endpoint takes a JSON body.
  • Input is file_path xor file_url. Stage your file with a PUT first, then reference it by the staged path (or pass a remote URL, gated behind AUDIOLLA_FETCH_MODE which defaults to disabled — the box doesn’t fetch arbitrary URLs unless you explicitly turn that on).
  • Output is output_path xor output_url. Enforced server-side by a shared validate_output_xor() helper called from every audio-producing handler. No more raw audio bytes riding home in the HTTP response — you get back JSON like {"path": "out/vocals.wav", "size": 4213880} and pull the actual file from GET /v1/files/{path}.
  • async_job=true on any endpoint fires the job into a background queue (in-memory, src/audiolla/jobs.py), auto-stages the result to jobs/{id}.{ext} if you didn’t set an explicit output, and — if you passed a webhook_url — POSTs the completed job payload back to you with retries when it’s done. No polling required if you don’t want to poll.

The file-staging path (src/audiolla/files.py) also isn’t naive about the fact that it’s taking arbitrary user-supplied path strings. It strips leading slashes, rejects null bytes and backslashes, rejects ./.. segments outright, then does the paranoid thing on top of the lexical check: resolves the candidate path and requires it to still be a child of the staging root, Path.resolve() and all. So “just concatenate the path and hope” isn’t the security model — actual traversal rejection is.

Thirty-five engines, and I counted every one of them myself

The project’s own README says “thirty audio engines” in the tagline. I didn’t trust it — READMEs rot the second nobody updates them after adding a feature — so I opened engines.json and counted the top-level keys myself. The real number on the CUDA build is thirty-five. The CPU-only build (engines-cpu.json) drops to twenty-seven, because eight of them — three extra Demucs variants and all five text-to-audio generators — simply aren’t listed in the CPU config at all. Only six of those eight are actually marked cuda_only: true in the registry (htdemucs_ft plus all five generators); htdemucs_6s and mdx_extra carry no such flag, they’re just absent from the CPU build’s engine list. Here’s the full 35, grouped by what they actually do, straight from the registry:

Stem separation (4)

  • htdemucs — Hybrid Transformer Demucs, 4 stems, the default balance of speed and quality
  • htdemucs_ft — fine-tuned variant, highest quality, ~4x slower, CUDA-only at usable speed
  • htdemucs_6s — 6-stem experimental variant, adds guitar and piano
  • mdx_extra — MDX-Net, MUSDB-trained, strong on vocal isolation

UVR-based restoration (5)

  • uvr-dereverb — BS-Roformer de-reverb
  • uvr-deecho — VR Architecture de-echo, normal and aggressive modes
  • uvr-denoise — MelBand Roformer AI denoise (SDR 28)
  • uvr-karaoke — MelBand Roformer, strips lead vocal, keeps the backing track
  • uvr-vocal-bsr — BS-Roformer vocal/instrumental split (SDR 13)

Mastering and DSP (4)

  • matchering — reference-based automatic mastering (GPL v3)
  • pedalboard-chain — preset mastering chain, transparent/loud presets (GPL v3)
  • sox-transform — SoX DSP chain: gain, EQ, compressor, reverb, pitch, tempo
  • fx-chain — generic ordered effects chain against pedalboard’s full allowlist, which I counted directly in fx_chain.py: 24 effect types (Compressor, Limiter, NoiseGate, Reverb, Chorus, Delay, Phaser, PitchShift, Convolution, and a bunch of filters/compressors more), not the 23 the README claims

MIDI (2)

  • midi-compose — JSON-to-MIDI transcoder (tracks + notes spec → Standard MIDI File)
  • midi-render — MIDI-to-audio via fluidsynth + the FluidR3_GM SoundFont

Transcription and analysis (7)

  • basic-pitch — Spotify’s polyphonic audio-to-MIDI transcription, ONNX backend so no TensorFlow drags along for the ride
  • librosa-analyze — BPM, key, loudness, duration, spectral features
  • chord-detect — chord + key detection via Krumhansl-Schmuckler and chroma template matching
  • hpss — harmonic/percussive split via librosa median filtering
  • stretch — time-stretch and pitch-shift via phase vocoder, independent tempo and semitone control
  • silence-detect — ffmpeg silencedetect, locate gaps, optional auto-trim
  • audio-fingerprint — Chromaprint fingerprinting via fpcalc, AcoustID-compatible

Speech (4)

  • deepfilter — DeepFilterNet DF3 neural speech and vocal enhancement
  • silero-vad — voice activity detection with timestamped segments
  • pyannote — speaker diarization via pyannote/speaker-diarization-3.1, needs a HUGGINGFACE_TOKEN
  • noise-reduce — spectral noise reduction, stationary and adaptive modes, no GPU required

Visualization, tagging, embeddings, metadata (4)

  • ffmpeg-render — spectrogram/waveform PNGs plus animated video across 8 filter modes
  • ast-tag — AudioSet tagging via Audio Spectrogram Transformer
  • clap-embed — 512-dim LAION CLAP embeddings with optional text-similarity query
  • metadata — ID3/Vorbis/FLAC tag read+write via mutagen

Text-to-audio generation, CUDA-only (5)

  • stable-audio-open — Stability Stable Audio Open 1.0, 47-second cap, no vocals, ~12GB VRAM at fp16, Stability Community License
  • musicgen-small — Meta MusicGen 300M, instrumental only, 30s cap, CC-BY-NC — server needs AUDIOLLA_ENABLE_NONCOMMERCIAL=1 set explicitly or the engine refuses to load
  • musicgen-medium — Meta MusicGen 1.5B, same license gate, better quality
  • riffusion — Stable Diffusion fine-tune that generates spectrograms and Griffin-Lim’s them back to audio, lo-fi character, CreativeML-OpenRAIL-M
  • audioldm2 — general SFX/ambience/foley generation, CC-BY 4.0, no license gate needed

Add it up: 4 + 5 + 4 + 2 + 7 + 4 + 4 + 5 = 35 (stem separation, restoration, mastering/DSP, MIDI, transcription/analysis, speech, visualization/tagging/metadata, text-to-audio generation, in that order). That’s the count I got straight from engines.json, not from marketing copy. The MusicGen license gate in particular is worth calling out because it’s an actual runtime guard, not just a comment — src/audiolla/engines/music_gen.py raises a hard error if you try to hit either MusicGen route without the noncommercial env var explicitly set, so nobody accidentally ships CC-BY-NC weights inside a commercial product because they didn’t read a warning label.

And then there’s everything that isn’t a “model” — 90 routes total

Engines are the ML/DSP heavy hitters, but the API surface is bigger than that — trim, mix, concat, speed, convert, fade, reverse, loop, BPM match, stereo width, split, pan, EQ, key match, sidechain duck, clip detect, mid/side encode/decode, beat slice, convolution reverb, transient shaping, multiband compression, remix, DJ prep (BPM + key + Camelot wheel + LUFS in one call), pitch correction, declip/dehum repair, loop-point detection, drum machine step sequencing, chords-to-MIDI, de-essing, stereo field analysis, thumbnailing, MIDI humanize/quantize/inspect/transform — pure ffmpeg/DSP ops that don’t need a model loaded at all. I pulled the exact route list out of openapi.yaml and parsed it: 90 distinct paths, 94 total operations once you count the ones that support more than one HTTP verb (like /v1/files/{path} handling GET, PUT, and DELETE). GET /v1/catalog hands you the whole thing back as a machine-readable, grouped-by-category list — built specifically so an agent (or you, half-awake) can discover what’s callable without reading the OpenAPI spec by hand.

Presets and pipelines — server-side chaining

You don’t have to hand-chain five HTTP calls every time you want to do the same five-step thing. audiolla ships three curated YAML workflow presets — I checked the presets/ directory directly, there are exactly three: master-for-spotify.yaml (three-band multiband compression crossed at 200Hz and 3kHz, normalized to -14 LUFS for streaming spec), podcast-cleanup.yaml (DeepFilterNet enhance → de-ess → normalize to -16 LUFS for Apple Podcasts), and vocal-cleanup.yaml (UVR de-reverb → UVR de-noise → de-ess → light pedalboard compression). Hit GET /v1/presets/{name} to see the step list, POST /v1/presets/{name} to run it against a staged file. And if the curated three don’t cover your case, POST /v1/pipeline lets you chain any of the byte-in/byte-out ops ad hoc, server-side, in one call — no bouncing intermediate files back and forth over HTTP.

Running It

# CPU, 27 engines
docker run --rm -it 
  -v $HOME/.audiolla-data:/data 
  -p 8000:8000 
  psyb0t/audiolla:latest
# GPU, all 35
docker run --rm -it --gpus all 
  -v $HOME/.audiolla-data:/data 
  -e AUDIOLLA_DEVICE=cuda 
  -p 8000:8000 
  psyb0t/audiolla:latest-cuda

And the actual v1.0 call shape — stage, process, retrieve:

# stage the input
curl -X PUT --data-binary @song.wav -H 'Content-Type: application/octet-stream' https://ciprian.51k.eu00/v1/files/uploads/song.wav
# rip the vocals
curl -X POST https://ciprian.51k.eu00/v1/audio/separate 
  -H 'Content-Type: application/json' 
  -d '{"file_path":"uploads/song.wav","engine":"htdemucs","stems":["vocals"],"output_path":"out/vocals.wav"}'
# pull the result back
curl -o vocals.wav https://ciprian.51k.eu00/v1/files/out/vocals.wav

No account, no per-minute billing, no vendor deciding tomorrow that vocal isolation now costs credits. Optional bearer-token auth (AUDIOLLA_AUTH_TOKEN) locks the whole API down except /healthz, using hmac.compare_digest() for the token check instead of a naive ==, because timing side channels on auth checks are the kind of thing that’s cheap to avoid and expensive to explain later.

MCP, and where this actually plugs in

Most of that REST surface mirrors onto an MCP server mounted at /v1/mcp — I counted 82 @mcp.tool() functions in src/audiolla/mcp_server.py against 90 OpenAPI paths, so it’s not a strict 1:1, but it’s enough tool calls that an LLM agent can drive stem separation, mastering, transcription, whatever, as native tool calls instead of you writing a bespoke adapter. Audio-producing MCP tools got the same v1.0 treatment as the REST side — the old audio_base64/midi_base64/image_base64 response modes are gone, because a 10MB WAV base64-encoded is roughly 3 million tokens of pure garbage for an LLM to choke on. Every generative tool now requires output_path or output_url, same xor rule as the REST API, and the agent gets back a tiny JSON pointer instead of a payload that blows the context window.

This is exactly the shape aigate is built to route to — audiolla is one of the backend services aigate can put behind its gateway, giving an agent audio production as a tool without the agent (or you) ever needing to know which of the thirty-five engines is actually doing the work underneath. It sits in the same self-hosted-infra family as docker-mediaproc (ffmpeg/sox/ImageMagick over SSH) and hybrids3 (self-hosted S3) — small, single-purpose Docker services that plug into a bigger self-hosted stack instead of demanding you build the whole stack around them.

Why I Actually Built It

I didn’t build audiolla because stem separation or MIDI transcription needed to exist — Demucs and basic-pitch already do their jobs fine on their own. I built it because I was done maintaining five venvs and a shell script duct-taping them together every time I wanted to do more than one thing to an audio file. Now it’s one container, one port, a JSON body, and thirty-five engines I counted myself sitting behind it, none of which are fighting each other over a torch version anymore. Grab it at github.com/psyb0t/docker-audiolla or off Docker Hub if you want it right now. Five venvs and a shell script is not a pipeline. This is.