aigate: Your Own Fucking AI Infrastructure

A 3060 and no cloud budget. AI models behind one endpoint — six cloud providers with free tiers, five local engines that run on your own hardware, the rest on flat-rate subs or pay-per-token as last resort.
Not in theory. Right now. aigate is a Docker stack behind a single nginx port. Text generation, image generation, speech synthesis, transcription, web search, browser automation, object storage, agentic code execution, sandboxed multi-language code execution, an email gateway, your Telegram account as an MCP tool, time-series forecasting, audio production, a video toolkit with lipsync, an async job queue, and a web UI. On hardware that costs less than one month of an OpenAI API bill.
The whole thing is OpenAI-compatible. Point any client at http://localhost:4000 and it works. Existing code, existing SDKs, existing tools — they all just talk to it like it’s OpenAI. It isn’t. It’s yours.

Models — Most of Them Free

Six providers — Groq, Cerebras, OpenRouter, HuggingFace, Mistral, Cohere — have free tiers. No credit card to start. But let’s be honest about what “free” means, because the marketing pages won’t be: free means rate-limited and capped, not unlimited. Groq gives you 30 RPM and somewhere between 1K and 14.4K requests/day depending on the model. Cerebras is 5 RPM / 30K TPM / 1M TPD across four models. OpenRouter is 50 requests/day at $0, 1K/day if you’ve ever put $10 in. HuggingFace hands you $0.10/month in credits — that’s an evaluation budget, not a tier. Cohere’s trial is 1K calls per month and it runs out fast. Mistral’s “Experiment” tier doesn’t publish exact limits at all.
So no, you’re not running a business on free tokens. But you are running your own workloads on them, and when one runs dry the next one picks up. That’s the whole point.
Two more — claudebox and pibox-zai — run on flat-rate subscriptions, so the marginal token costs nothing. Anthropic and OpenAI are in the stack but they’re last resort. Five local engines run on your own hardware — no network, no rate limits, no cost at all.
The routing philosophy is simple: never pay for a token you could get free.
LiteLLM handles it. You request groq-llama-3.3-70b. Groq rate-limits you. LiteLLM silently falls to cerebras-qwen3-235b. Cerebras is down. Falls to mistral-small. Mistral responds. You get the answer. The client never knew anything happened. Every model has a fallback chain — free cloud first, then flat-rate, then pay-per-token, then local. The chain is rebuilt every time you start the stack, filtered to only the providers you’ve actually enabled.

groq-llama-3.3-70b → 429 rate limited
  ↓ fallback
cerebras-qwen3-235b → 503 unavailable
  ↓ fallback
mistral-small → 200 ✓

You didn’t write retry logic. You didn’t write fallback logic. You sent one request and got one response.

The Architecture

nginx :4000                                    ┌──────────── always on ────────────┐
├─► /claudebox/            → claudebox         │ nginx, LiteLLM, PostgreSQL, Redis │
├─► /pibox-zai/            → pibox-zai         │ proxq — everything else is opt-in │
├─► /stealthy-auto-browse/ → HAProxy → [×5]    └───────────────────────────────────┘
├─► /storage/              → hybrids3
├─► /q/                    → proxq → LiteLLM (async, returns job ID)
├─► /librechat/            → LibreChat (web UI, LIBRECHAT=1)
├─► /searxng/              → SearXNG (meta-search, SEARXNG=1)
├─► /telethon/             → Telethon (Telegram client, TELETHON=1)
├─► /mailbox/              → mailbox (IMAP+SMTP gateway, MAILBOX=1)
├─► /piston/               → piston (sandboxed code execution, PISTON=1)
├─► /predictalot/          → predictalot (forecasting + tabular ML, PREDICTALOT=1)
├─► /audiolla/             → audiolla (audio production, AUDIOLLA=1)
├─► /flickies/             → flickies (video toolkit, FLICKIES=1)
└─► /                      → LiteLLM (sync)
    ├─ Groq              (free: 30 RPM, 1K-14.4K RPD per model, GROQ=1)
    ├─ Cerebras          (free: 5 RPM / 30K TPM / 1M TPD, 4 models, CEREBRAS=1)
    ├─ OpenRouter        (free: 50 RPD $0 / 1K RPD with $10+, OPENROUTER=1)
    ├─ HuggingFace       (free: $0.10/mo credits — eval only, HUGGINGFACE=1)
    ├─ Mistral           (free "Experiment" tier, MISTRAL=1)
    ├─ Cohere            (trial: 1K calls/MONTH, COHERE=1)
    ├─ Ollama            (local CPU + CUDA, OLLAMA=1 / OLLAMA_CUDA=1)
    ├─ Talkies           (local ASR + TTS, TALKIES=1 / TALKIES_CUDA=1)
    ├─ sd.cpp            (local image gen, SDCPP=1 / SDCPP_CUDA=1)
    ├─ vLLM              (local text LLM + embeddings, VLLM=1 / VLLM_CUDA=1)
    ├─ llama.cpp         (local GGUF + vision-VLM, LLAMACPP=1 / LLAMACPP_CUDA=1)
    ├─ claudebox         (flat-rate, CLAUDEBOX=1)
    ├─ pibox-zai         (flat-rate, PIBOX_ZAI=1)
    ├─ Anthropic         (pay-per-token, ANTHROPIC=1)
    └─ OpenAI            (pay-per-token, OPENAI=1)
MCP servers (all optional):
  ├─ stealthy_auto_browse  — run_script: multi-step browser automation (BROWSER=1)
  ├─ hybrids3              — upload, download, list, delete, presign (HYBRIDS3=1)
  ├─ claudebox             — agentic Claude Code via OAuth or API key (CLAUDEBOX=1)
  ├─ pibox_zai             — agentic pi-coding-agent via z.ai/GLM (PIBOX_ZAI=1)
  ├─ telethon              — your Telegram account as a tool (TELETHON=1)
  ├─ mailbox               — IMAP+SMTP across N accounts (MAILBOX=1)
  ├─ predictalot           — time-series forecasting (PREDICTALOT=1)
  ├─ audiolla              — audio production, MIR, mastering, MIDI (AUDIOLLA=1)
  ├─ flickies              — lipsync, face restore, ffmpeg ops (FLICKIES=1)
  └─ mcp_tools             — generate_image + generate_tts + search_web
                             + execute_code (auto-enabled)

Everything is opt-in. Flip flags in .env. Don’t have an Anthropic key? Don’t set it. Only have a CPU? Skip the CUDA flags. The stack adapts to what you’ve got and rebuilds its config accordingly. The always-on core is five things: nginx, LiteLLM, PostgreSQL, Redis, proxq. Everything else is a flag.

Tools Any Model Can Call

This is what makes aigate more than a proxy with a fallback chain. A whole pile of MCP servers, dozens of tools. Any model with function calling can invoke any of them autonomously. You prompt. The model decides what tools it needs. You get results.
The concrete workflow: you send a prompt to Groq — free, fast. The model decides it needs to research something. It calls search_web — SearXNG queries Google, Bing, DuckDuckGo simultaneously, returns results. The model needs more detail on one result. It calls the browser tool. A Camoufox instance opens real Firefox, moves a real mouse cursor, loads the page, extracts content. The model reads it, decides to save the result. Calls the storage tool. hybrids3 writes the file, returns a public URL. The model decides it needs to generate an image for the report. Calls generate_image. stable-diffusion.cpp renders it locally, uploads to storage, returns a URL. The model hands you a structured answer with links to everything it produced. One API call. Zero tokens paid. The client saw one request and one response.

stealthy_auto_browse

Five Camoufox replicas behind HAProxy. Camoufox is hardened Firefox — real OS-level mouse and keyboard input via PyAutoGUI, zero CDP exposure, persistent fingerprints per session. Passes Cloudflare. Passes CreepJS. Passes BrowserScan. Passes Pixelscan. Not “mostly passes” — actually passes, because it’s not detectable as automation in the ways those systems check.
One tool: run_script. Multi-step scripts — navigate, click, type, extract, screenshot, scroll, wait for elements, execute JavaScript. The model writes the script, the browser executes it, you get back structured data from the live page.

hybrids3

S3-compatible object storage running locally. The uploads bucket is public-read — files accessible by direct URL without signing. Presigned PUT URLs for direct uploads. Auto-expiry. Tools for put, get, list, delete, info, presign, list buckets.
This solves a specific problem in agentic workflows. When a model produces something large — a scraped dataset, a generated image, a rendered report — you don’t stuff it into the context window. You put it in storage, get a URL, pass the URL. The next step can fetch it. You can fetch it. Context stays clean.

Two Coding Agents — claudebox and pibox-zai

Two different agentic coding agents, not two copies of the same one. claudebox runs Claude Code on your subscription or API key. pibox-zai runs pi-coding-agent pointed at z.ai for GLM models. Both get full shell access, persistent workspaces, file I/O, tool use — and both expose a REST API, an OpenAI-compatible endpoint, and an MCP server.
You’re using Groq for speed. Groq hits something that needs deep coding work. Groq calls the claudebox tool. Claude Code picks it up, gets a shell, writes code, runs tests, returns structured results. Back in Groq’s context. The orchestration happens inside the model’s function calling loop — you didn’t write any of that logic.

piston — Sandboxed Code Execution

LLMs are catastrophically bad at arithmetic and hashing and parsing. So don’t make them guess. execute_code runs actual code in an nsjail sandbox — own user namespace, chroot, seccomp filter, cgroup limits, no network — and hands the model a deterministic result.
Default install is Python + Node. Add Bash, Deno, Go, Rust, TypeScript, or 40-odd other languages via PISTON_LANGUAGES and rebuild. The model needs a SHA-256, a date diff, a regex applied to 10K rows? It writes code, piston runs it in a jail, the number comes back correct instead of hallucinated.

mcp_tools

generate_image, generate_tts, search_web, and execute_code. Auto-enabled when the matching backend is active. All return structured JSON — generated files are uploaded to hybrids3 automatically with persistent URLs. No base64 blobs in the context window.
Image generation routes through FLUX, DALL-E, or stable-diffusion.cpp depending on what’s enabled. TTS routes through Kokoro (CPU + CUDA), Qwen3-TTS with voice cloning, voice design and emotion control (CUDA), or OpenAI TTS. Web search queries SearXNG. Code execution goes to piston. The tools discover available models dynamically from LiteLLM — they always reflect what’s actually running.

telethon

Your actual Telegram account as a tool. Not the Bot API — full MTProto, same access you have on your phone. Read messages, send messages, list dialogs, manage groups, forward content, edit, delete, mark read, send files. The model decides when. The agent acts as you.
A free-tier Groq model that searches the web, scrapes a page, generates an image, and sends the result to your Saved Messages — zero tokens paid, one conversation, multiple tool calls. Or a cron job that summarizes your work group’s last 24 hours into a digest and DMs it to you every morning. Your account, programmable.
Powered by telethon-plus. Flip TELETHON=1, set TELETHON_API_ID / TELETHON_API_HASH / TELETHON_SESSION in .env. The session string is full account access — treat it like the password it effectively is.

mailbox — Email as a Tool

Stateless IMAP + SMTP across N accounts from one YAML config. Unified inbox, per-account list/search/CRUD, SMTP send. REST API plus a flat MCP tool set where a mailbox parameter picks the account.
Which means a model can read your inbox, find the invoice, extract the total, and reply — without you writing a single line of IMAP handling. MAILBOX=1, point MAILBOX_CONFIG at your YAML, done.

predictalot — Forecasting and Tabular ML

Five zero-shot time-series foundation models — chronos-2, timesfm-2.5, moirai-2, toto-1, sundial-base-128m — across six forecast types (univariate, multivariate, past covariates, future covariates, samples, and per-type weighted ensembles) at /v1/timeseries/<type>/….
Then a whole sibling family at /v1/tabular/*: nine supervised backends (lightgbm, xgboost, hist-gbt, random-forest, logistic, mlp, svm-rbf, knn, naive-bayes) and three meta-learners (calibrated, stacking, diversified). 26 MCP tools cover the foundation models; tabular is REST-only. CPU or CUDA.
Zero-shot means no training run. You hand it a series, it forecasts. That’s it.

audiolla — Audio Production

This one’s absurd in scope. Stem separation (Demucs / UVR). Restoration — de-reverb, de-echo, de-noise. Mastering via matchering and pedalboard chains, with curated presets like master-for-spotify, podcast-cleanup, vocal-cleanup. MIR analysis: BPM, key, LUFS, beats, onsets, melody, chords, segments. DSP transforms through sox and ffmpeg. Loudness normalization. Speech enhancement (DeepFilterNet). VAD (silero). Diarization (pyannote). CLAP embeddings and zero-shot audio classification. AudioSet tagging. Audio→MIDI via basic-pitch, and MIDI compose / inspect / transform / render through fluidsynth.
Plus text-to-audio generation — stable-audio-open, musicgen, riffusion, audioldm2 — CUDA only.
Async jobs and webhooks throughout. The v1.0+ contract is strict: JSON body on every audio endpoint, raw bytes only at PUT /v1/files/{path}, and output_path xor output_url mandatory on anything that produces audio.

flickies — Video and Lipsync

Lipsync via LatentSync 1.5 (ByteDance, Apache-2.0, ~8 GB VRAM, the default on CUDA) and Wav2Lip / Wav2Lip-GAN (LRS2, non-commercial, gated). Face restore via GFPGAN v1.4. ffmpeg ops — trim, concat, transcode including gif/fps/codec changes, scale, mux audio, extract audio, thumbnail grid. ffprobe info. Async jobs and webhooks. 11 MCP tools, same JSON-body and output_path xor output_url contract as audiolla.
GFPGAN and LatentSync 1.5 are CUDA-only. The CPU image still does ffmpeg ops and a very slow Wav2Lip.

Local Inference — Slower, But Yours

Five local engines. Every one of them has a CPU mode.
This is the “poor people’s computer” part. Your old laptop with 16GB of RAM can run text generation, image generation, transcription, and speech synthesis. No API key. No network. No rate limit. No bill. It’s slower than cloud. Sometimes much slower. But it’s yours, it’s private, and it works offline.
Ollama — llama3.2:3b, qwen3:4b, smollm2:1.7b, qwen2.5-coder:1.5b, qwen2.5-coder:3b, phi4-mini (reasoning, 128K context), gemma4:e2b (vision), gemma3:4b (vision), nuextract-v1.5 (structured text → JSON extraction), dolphin-phi, plus two embedding models for RAG. The smallest needs 1GB of RAM. OLLAMA_CUDA=1 adds the heavier set: qwen3:8b, gemma4:e4b (vision), deepseek-coder-v2:16b (MoE, 160K context), deepseek-r1:8b (reasoning), qwen3-abliterated:16b (uncensored), gemma4-abliterated:e4b (uncensored vision), qwen2.5-coder:7b, llama3.1:8b. Flash attention, quantized KV cache, shared model storage with the CPU service — no duplicate downloads.
talkies — the unified ASR + TTS service, and the replacement for what used to be Speaches in this stack. CPU: whisper-large-v3 and -turbo, canary-180m-flash, nemotron-3.5-asr-0.6b for transcription; Kokoro for TTS. CUDA adds parakeet-tdt-0.6b-v3, canary-1b-flash, canary-qwen-2.5b, and the Qwen3-TTS family — 0.6B and 1.7B, with voice cloning, voice design, and emotion control through the instructions field. OpenAI-compatible endpoints, so existing Whisper calls work unchanged. TALKIES=1 / TALKIES_CUDA=1.
stable-diffusion.cpp — local image generation. CPU runs sd-turbo and sdxl-turbo out of the box. SDCPP_CUDA=1 for hardware acceleration and the full model set: sd-turbo, sdxl-turbo, sdxl-lightning, flux-schnell, juggernaut-xi. Models download on first use and cache locally. The OpenAI-compatible /images/generations endpoint means existing code works as-is.
vLLM — local text LLMs plus embeddings. qwen3-0.6b for generation, nomic-embed-v2 for embeddings. VLLM=1 / VLLM_CUDA=1.
llama.cpp — GGUF models and vision-VLMs, including Surya OCR 2 for document understanding. LLAMACPP=1 / LLAMACPP_CUDA=1.
Local models sit at the end of the fallback chain by default. Cloud fails? Local picks it up. Or you target them directly: "model": "local-ollama-cpu-llama3.2-3b". Zero network. Zero cost. Slower, but it answers.

One GPU, Everything

Here’s the engineering problem: you have one GPU. Ollama wants VRAM for the LLM. sd.cpp wants VRAM for image generation. talkies wants VRAM for transcription and speech. audiolla wants it for stem separation. flickies wants ~8GB of it for LatentSync. Load them all and you OOM.
The resource manager solves this automatically. A LiteLLM callback enforces mutual exclusion per hardware — one CUDA job at a time. When an image generation request arrives while an LLM is loaded, the resource manager acquires the semaphore, unloads the LLM, then lets the image generation proceed. When a TTS request comes after that, it unloads the image generator first. Same logic on CPU.
Every service has its own unload API and the resource manager knows all of them: Ollama takes keep_alive: 0, sd.cpp has POST /sdcpp/v1/unload, talkies / vllm-cuda / llamacpp-cuda take DELETE /api/ps/{model_id} per model or POST /unload to kill whatever’s loaded, and audiolla has POST /v1/unload to bulk-evict every loaded engine.
There are operator-facing endpoints too, when you want to nuke it all by hand: POST /v1/unload/cuda fans out concurrently to ollama-cuda, sdcpp-cuda, talkies-cuda, vllm-cuda, llamacpp-cuda, audiolla-cuda and flickies-cuda. POST /v1/unload/cpu does the same seven on the CPU side. POST /v1/unload runs both in sequence and hands you back a per-service report of what actually got evicted.
You don’t manage any of this. You send requests. The platform juggles VRAM automatically. The only cost is latency — the first request after a swap includes model load time. After that, it’s fast until the idle timeout unloads the model to free memory for the next thing.

Web Search

SearXNG at /searxng/. Self-hosted meta-search — queries Google, Bing, DuckDuckGo, and Wikipedia simultaneously. No API key. Runs entirely locally.
The MCP search_web tool means any function-calling model can search the web autonomously. The model decides it needs to look something up, calls the tool, gets results, continues reasoning. You didn’t build a search integration. You flipped SEARXNG=1.

LibreChat — The Daily Driver

Everything above works from the API. But for daily use, there’s LibreChat at /librechat/.
All models in the dropdown. All MCP tools wired in. Pick a model, start talking. The model can still invoke the browser, storage, the coding agents, image generation, TTS, and web search autonomously — everything available from the API is available in the UI. Conversation history backed by MongoDB. File uploads. WebSocket streaming.
First registered user becomes admin. Set LIBRECHAT_ALLOW_REGISTRATION=false after that and you’re the only one in.
Enable it: LIBRECHAT=1 in .env.

Async Queue

Long inference requests time out. Hit /q/ instead of / and the request goes into a Redis-backed queue. You get a job ID back instantly. The actual inference runs in the background. Poll for status, fetch the result when it’s done.

# submit — returns 202 immediately
curl http://localhost:4000/q/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model": "cerebras-qwen3-235b", "messages": [{"role":"user","content":"write a novel"}]}'
# → {"jobId": "550e8400-e29b-41d4-a716-446655440000"}
# check status
curl http://localhost:4000/q/__jobs/550e8400-e29b-41d4-a716-446655440000 \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"
# get the result
curl http://localhost:4000/q/__jobs/550e8400-e29b-41d4-a716-446655440000/content \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

Configurable concurrency, retention, timeouts, retries, and response caching. Only OpenAI API paths get queued — health checks and admin requests pass through directly.

It Ships Its Own Agent Skill

The repo carries .agents/skills/aigate/ — a ClawHub-published agent skill that teaches a coding agent how to drive the stack. So instead of explaining your own infrastructure to an assistant every session, you install the skill and it already knows the endpoints, the flags, and the tool contracts.

Security

Internal services — PostgreSQL, MongoDB, Redis, the browser cluster, the storage backend — have no host ports. They’re on isolated Docker networks. Nothing reaches them from outside the stack. The only exposed surface is nginx on port 4000, and that requires bearer token auth.
Application containers run with no-new-privileges:true. Two carve-outs, both deliberate and both documented in the compose file: piston needs privileged: true because nsjail has to build user namespaces and chroots per execution — the actual isolation lives inside the nsjail subprocess, and the privileged container is what nsjail needs in order to construct that jail. tailscale needs NET_ADMIN for its userspace tun device. Everything else drops privileges. make run validates that any file paths in .env actually exist before starting — no silently broken volume mounts.
Want it publicly reachable without opening a firewall port? CLOUDFLARED=1 — Cloudflare Tunnel. DDoS protection, TLS termination, no open ports, no IP to scan. Quick tunnel for a random *.trycloudflare.com URL, or a named tunnel for a fixed domain.
Don’t want it public at all, but still want to reach it from your laptop, phone, or another box? TAILSCALE=1 with TS_AUTHKEY=tskey-auth-... and a TS_HOSTNAME. A Tailscale sidecar joins your tailnet and runs tailscale serve in L4 TCP forwarding mode straight to nginx:4000 — no Host-header matching, no FQDN configuration on the tailscale side, no HTTPS auto-cert (TLS termination, if you want it, lives in nginx). nginx receives the original request bytes as-is and routes them through its existing vhost/path logic. Access via http://<TS_HOSTNAME>.<tailnet>.ts.net — plain HTTP is fine here because WireGuard already encrypts every byte inside the tailnet. Works with hosted Tailscale or self-hosted Headscale (use TS_EXTRA_ARGS=--login-server=...). State persists in .data/tailscale/ so reboots reuse the existing login. Combine it with the bearer token auth and you’ve got two completely independent layers of access control.
Hundreds of tests. Health checks, routing, auth, MCP tool validation, storage CRUD, browser automation, agentic coding runs, async job lifecycle, local TTS/STT round-trips, CUDA resource manager verification, local image generation, LLM-to-MCP end-to-end tool calling, SearXNG search, Telethon MTProto round-trips against a real account. Plus security: cross-token isolation, session hijack attempts, HTTP request smuggling (CL.TE/TE.CL), h2c smuggling, SSRF via browser and MCP to internal services, prompt injection key extraction, path traversal, S3 presign abuse, stored XSS, model name injection, header injection, Docker socket isolation. This isn’t a hobby project test suite. This is paranoia as a feature.

Setup

git clone https://github.com/psyb0t/aigate && cd aigate
cp .env.example .env  # edit: add keys, flip flags
make run-bg

Every variable is documented in .env.example. Enable what you have, ignore what you don’t.
If resources matter — and on a normal computer they do — make limits reads your available RAM and CPU and writes recommended limits for every service. MAXUSE=80 make limits caps the whole stack at 80% of system resources if you’re sharing the machine with other workloads. CUDA services are resource-manager-aware, so the budget counts the largest one — not every GPU service at full allocation simultaneously.

# free tier, auto-fallback
curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model": "cerebras-qwen3-235b", "messages": [{"role":"user","content":"hello"}]}'
# local, no network, no limits
curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model": "local-ollama-cpu-llama3.2-3b", "messages": [{"role":"user","content":"hello"}]}'
# image generation
curl http://localhost:4000/images/generations \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model": "hf-flux-schnell", "prompt": "a cat riding a skateboard"}'
# local image generation (no network, no cost)
curl http://localhost:4000/images/generations \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model": "local-sdcpp-cpu-sd-turbo", "prompt": "a red panda in a forest", "size": "512x512"}'
# transcription (local, CPU)
curl http://localhost:4000/audio/transcriptions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -F "model=local-talkies-whisper-large-v3-turbo" -F "[email protected]"
# text-to-speech (local, multiple voices)
curl http://localhost:4000/audio/speech \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model": "local-talkies-kokoro-tts", "input": "Hello world", "voice": "af_heart"}' \
  -o speech.mp3
# web search (no API key, self-hosted)
curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model": "groq-qwen3-32b", "messages": [{"role":"user","content":"search the web for latest rust release notes"}]}'
# async — submit and poll
curl http://localhost:4000/q/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model": "cerebras-qwen3-235b", "messages": [{"role":"user","content":"write a novel"}]}'

Six providers with free tiers, five local engines, the rest as fallback. Slower on a normal computer — but it runs, it’s private, and nobody can rate-limit you out of your own infrastructure.
github.com/psyb0t/aigate