A 3060 and no cloud budget. AI models behind one endpoint — five cloud providers with free tiers, five local engines that run on your own hardware, the rest on subscription plans or pay-per-token as last resort.
Not in theory. Right now. aigate is a Docker stack behind a single nginx port. Text generation, image generation, speech synthesis, transcription, web search, browser automation, object storage, agentic code execution, sandboxed multi-language code execution, an email gateway, your Telegram account as an MCP tool, time-series forecasting, typed yes/no and multiple-choice decisions, audio production, a video toolkit with lipsync, an async job queue, and a web UI. On hardware that costs less than one month of an OpenAI API bill.
The whole thing is OpenAI-compatible. Point any client at http://localhost:4000 and it works. Existing code, existing SDKs, existing tools — they all just talk to it like it’s OpenAI. It isn’t. It’s yours.
Models — Most of Them Free
Five providers — Groq, OpenRouter, HuggingFace, Mistral, Cohere — have free tiers. No credit card to start. But let’s be honest about what “free” means, because the marketing pages won’t be: free means rate-limited and capped, not unlimited. Groq gives you 30 RPM and somewhere between 1K and 14.4K requests/day depending on the model. OpenRouter is 50 requests/day at $0, 1K/day if you’ve ever put $10 in. HuggingFace hands you $0.10/month in credits — that’s an evaluation budget, not a tier. Cohere’s trial is 1K calls per month and it runs out fast. Mistral’s “Experiment” tier doesn’t publish exact limits at all. Cerebras used to be on this list. It isn’t any more: as of the last audit every model on it answers “payment required”, so it’s in the stack as a paid provider with three models, not a free tier. That’s the other thing about free tiers, they get withdrawn, and a stack built on exactly one of them is a stack that breaks the day it goes.
So no, you’re not running a business on free tokens. But you are running your own workloads on them, and when one runs dry the next one picks up. That’s the whole point.
Three more — claudebox, pibox-zai and pibox — are agent containers. The first two run on subscription plans, which is not the same as free: the allowance is metered, you just pay for it monthly instead of per token. Anthropic and OpenAI are in the stack but they’re last resort. Five local engines run on your own hardware — no network, no rate limits, no cost at all.
The routing philosophy is simple: never pay for a token you could get free.
LiteLLM handles it. You request groq-gpt-oss-120b. Groq rate-limits you. LiteLLM silently falls to cerebras-gpt-oss-120b. Cerebras is down. Falls to mistral-large. Mistral responds. You get the answer. The client never knew anything happened. Most models have a fallback chain: free cloud first, then subscription, then pay-per-token, and the speech, image, embedding and coding chains keep a local model in there too. The chain is rebuilt every time you start the stack, filtered to only the providers you’ve actually enabled.
groq-gpt-oss-120b → 429 rate limited
↓ fallback
cerebras-gpt-oss-120b → 503 unavailable
↓ fallback
mistral-large → 200 ✓You didn’t write retry logic. You didn’t write fallback logic. You sent one request and got one response.
The Architecture
nginx :4000 ┌──────────── always on ────────────┐
├─► /claudebox/ → claudebox │ nginx, LiteLLM, PostgreSQL, Redis │
├─► /pibox-zai/ → pibox-zai │ proxq — everything else is opt-in │
├─► /pibox/ → pibox └───────────────────────────────────┘
├─► /stealthy-auto-browse/ → HAProxy → [×5]
├─► /storage/ → hybrids3
├─► /q/ → proxq → LiteLLM (async, returns job ID)
├─► /librechat/ → LibreChat (web UI, LIBRECHAT=1)
├─► /searxng/ → SearXNG (meta-search, SEARXNG=1)
├─► /telethon/ → Telethon (Telegram client, TELETHON=1)
├─► /mailbox/ → mailbox (IMAP+SMTP gateway, MAILBOX=1)
├─► /piston/ → piston (sandboxed code execution, PISTON=1)
├─► /predictalot/ → predictalot (forecasting + tabular ML, PREDICTALOT=1)
├─► /decidealot/ → decidealot (typed decisions, DECIDEALOT=1)
├─► /audiolla/ → audiolla (audio production, AUDIOLLA=1)
├─► /flickies/ → flickies (video toolkit, FLICKIES=1)
└─► / → LiteLLM (sync)
├─ Groq (free: 30 RPM, 1K-14.4K RPD per model, GROQ=1)
├─ Cerebras (paid plan, 3 models, CEREBRAS=1)
├─ OpenRouter (free: 50 RPD $0 / 1K RPD with $10+, OPENROUTER=1)
├─ HuggingFace (free: $0.10/mo credits — eval only, HUGGINGFACE=1)
├─ Mistral (free "Experiment" tier, MISTRAL=1)
├─ Cohere (trial: 1K calls/MONTH, COHERE=1)
├─ Ollama (local CPU + CUDA, OLLAMA=1 / OLLAMA_CUDA=1)
├─ Talkies (local ASR + TTS, TALKIES=1 / TALKIES_CUDA=1)
├─ sd.cpp (local image gen, SDCPP=1 / SDCPP_CUDA=1)
├─ vLLM (local text LLM + embeddings, VLLM=1 / VLLM_CUDA=1)
├─ llama.cpp (local GGUF VLM + embeddings, LLAMACPP=1 / LLAMACPP_CUDA=1)
├─ claudebox (subscription, CLAUDEBOX=1)
├─ pibox-zai (subscription, PIBOX_ZAI=1)
├─ Anthropic (pay-per-token, ANTHROPIC=1)
└─ OpenAI (pay-per-token, OPENAI=1)
MCP servers (all optional):
├─ stealthy_auto_browse — run_script: multi-step browser automation (BROWSER=1)
├─ hybrids3 — upload, download, list, delete, presign (HYBRIDS3=1)
├─ claudebox — agentic Claude Code via OAuth or API key (CLAUDEBOX=1)
├─ pibox_zai — agentic pi-coding-agent via z.ai/GLM (PIBOX_ZAI=1)
├─ pibox — agentic pi-coding-agent on your own models (PIBOX=1,
direct at /pibox/mcp/ only, not in the aggregated /mcp/)
├─ telethon — your Telegram account as a tool (TELETHON=1)
├─ mailbox — IMAP+SMTP across N accounts (MAILBOX=1)
├─ predictalot — time-series forecasting (PREDICTALOT=1)
├─ decidealot — typed choice / score / yes-no decisions (DECIDEALOT=1)
├─ audiolla — audio production, MIR, mastering, MIDI (AUDIOLLA=1)
├─ flickies — lipsync, face restore, ffmpeg ops (FLICKIES=1)
└─ mcp_tools — generate_image + generate_tts + search_web
+ execute_code (auto-enabled)Everything is opt-in. Flip flags in .env. Don’t have an Anthropic key? Don’t set it. Only have a CPU? Skip the CUDA flags. The stack adapts to what you’ve got and rebuilds its config accordingly. The always-on core is five things: nginx, LiteLLM, PostgreSQL, Redis, proxq. Everything else is a flag.
Tools Any Model Can Call
This is what makes aigate more than a proxy with a fallback chain. A whole pile of MCP servers, dozens of tools. All of them served from one aggregated MCP endpoint. Hook LibreChat or any MCP-capable client or agent to it and any model with function calling can drive every one of them. You prompt. The model decides what tools it needs. You get results.
The concrete workflow: you’re in LibreChat, or any MCP-capable agent, and you send a prompt to Groq — free, fast. The model decides it needs to research something. It calls search_web — SearXNG queries Google, Bing, DuckDuckGo simultaneously, returns results. The model needs more detail on one result. It calls the browser tool. A Camoufox instance opens real Firefox, moves a real mouse cursor, loads the page, extracts content. The model reads it, decides to save the result. Calls the storage tool. hybrids3 writes the file, returns a public URL. The model decides it needs to generate an image for the report. Calls generate_image. stable-diffusion.cpp renders it locally, uploads to storage, returns a URL. The model hands you a structured answer with links to everything it produced. Zero tokens paid. The client runs the tool loop, aigate serves every tool in it. A bare /chat/completions call gets none of this, nothing attaches the tools for you.
stealthy_auto_browse
Five Camoufox replicas behind HAProxy. Camoufox is hardened Firefox — real OS-level mouse and keyboard input via PyAutoGUI, zero CDP exposure, persistent fingerprints per session. Passes Cloudflare. Passes CreepJS. Passes BrowserScan. Passes Pixelscan. Not “mostly passes” — actually passes, because it’s not detectable as automation in the ways those systems check.
One tool: run_script. Multi-step scripts — navigate, click, type, extract, screenshot, scroll, wait for elements, execute JavaScript. The model writes the script, the browser executes it, you get back structured data from the live page.
hybrids3
S3-compatible object storage running locally. The uploads bucket is public-read — files accessible by direct URL without signing. Presigned PUT URLs for direct uploads. Auto-expiry. Tools for put, get, list, delete, info, presign, list buckets.
This solves a specific problem in agentic workflows. When a model produces something large — a scraped dataset, a generated image, a rendered report — you don’t stuff it into the context window. You put it in storage, get a URL, pass the URL. The next step can fetch it. You can fetch it. Context stays clean.
Three Coding Agents — claudebox, pibox-zai and pibox
Three different agentic coding agents, not three copies of the same one. claudebox runs Claude Code on your subscription or API key. pibox-zai runs pi-coding-agent pointed at z.ai for GLM models (glm-5.3 and glm-5.3-flash). pibox is the same agent pointed back at this stack’s own LiteLLM, so it codes on whatever you already enabled: name the models with PIBOX_MODELS, pick a default with PIBOX_DEFAULT_MODEL. Two caveats there, and both bite: a model you list has to actually support tool calling or the agent has nothing to drive, and do not list an agent’s own model back to itself unless you enjoy recursion. All three get full shell access, persistent workspaces, file I/O, tool use — and all three expose a REST API, an OpenAI-compatible endpoint, and an MCP server. Streaming chat completions on all three can also carry the agent’s raw event records: send "stream_options": {"include_aicodebox_events": true} and named aicodebox.native SSE events ride along next to the normal OpenAI chunks, so you can watch the agent work instead of just waiting for its answer.
You’re using Groq for speed. Groq hits something that needs deep coding work. Groq calls the claudebox tool. Claude Code picks it up, gets a shell, writes code, runs tests, returns structured results. Back in Groq’s context. The orchestration happens inside the model’s function calling loop — you didn’t write any of that logic.
piston — Sandboxed Code Execution
LLMs are catastrophically bad at arithmetic and hashing and parsing. So don’t make them guess. execute_code runs actual code in an nsjail sandbox — own user namespace, chroot, seccomp filter, cgroup limits, no network — and hands the model a deterministic result.
Default install is Python + Node. Add Bash, Deno, Go, Rust, TypeScript, or 40-odd other languages via PISTON_LANGUAGES and rebuild. The model needs a SHA-256, a date diff, a regex applied to 10K rows? It writes code, piston runs it in a jail, the number comes back correct instead of hallucinated.
mcp_tools
generate_image, generate_tts, search_web, and execute_code. Auto-enabled when the matching backend is active, with one catch: PISTON=1 on its own doesn’t start this tool server, so execute_code only shows up once an image, TTS or search backend is on too. All return structured JSON — generated files are uploaded to hybrids3 automatically and come back as public URLs, which last until the uploads bucket’s TTL wipes them, a week by default. No base64 blobs in the context window.
Image generation goes local first: stable-diffusion.cpp with flux-schnell on CUDA or sd-turbo on CPU, and a hosted model like DALL-E only when nothing local is enabled. A hosted default breaks every single call the day the provider drops the model, and that’s exactly what HuggingFace did to FLUX. TTS routes through Kokoro (CPU + CUDA), Qwen3-TTS with voice cloning, voice design and emotion control (CUDA), Chatterbox Turbo (CUDA), or OpenAI TTS. Web search queries SearXNG. Code execution goes to piston. The tools discover available models dynamically from LiteLLM — they always reflect what’s actually running.
telethon
Your actual Telegram account as a tool. Not the Bot API — full MTProto, same access you have on your phone. Read messages, send messages, list dialogs, manage groups, forward content, edit, delete, mark read, send files. The model decides when. The agent acts as you.
A free-tier Groq model that searches the web, scrapes a page, generates an image, and sends the result to your Saved Messages — zero tokens paid, one conversation, multiple tool calls. Or a cron job that summarizes your work group’s last 24 hours into a digest and DMs it to you every morning. Your account, programmable.
Powered by telethon-plus. Flip TELETHON=1, set TELETHON_API_ID / TELETHON_API_HASH / TELETHON_SESSION in .env. The session string is full account access — treat it like the password it effectively is.
mailbox — Email as a Tool
Stateless IMAP + SMTP across N accounts from one YAML config. Unified inbox, per-account list, search, read, mark-seen and delete, SMTP send. REST API plus a flat MCP tool set where a mailbox parameter picks the account.
Which means a model can read your inbox, find the invoice, extract the total, and reply — without you writing a single line of IMAP handling. MAILBOX=1, point MAILBOX_CONFIG at your YAML, done.
predictalot — Forecasting and Tabular ML
Five zero-shot time-series foundation models — chronos-2, timesfm-2.5, moirai-2, toto-1, sundial-base-128m — across six forecast types (univariate, multivariate, past covariates, future covariates, past plus future covariates, and samples, each with a weighted ensemble on top) at /v1/timeseries/<type>/….
Then a whole sibling family at /v1/tabular/*: nine supervised backends (lightgbm, xgboost, hist-gbt, random-forest, logistic, mlp, svm-rbf, knn, naive-bayes) and three meta-learners (calibrated, stacking, diversified). 27 MCP tools, 26 for the foundation models plus unload_models; tabular is REST-only. CPU or CUDA.
Zero-shot means no training run. You hand it a series, it forecasts. That’s it.
decidealot: Typed Decisions, No Essays
The offline answer to TypeSafe’s Jev. Three local decision models, Laya, Von and CLM, all on by default, behind the System One API: send a state and a few named questions, each one a choice, a score or a noul yes/no, and you get typed answers back with real probabilities on them. No prose to parse, no “Sure! Here’s the JSON”, and the thing being judged never leaves your box. Fourteen local model selectors, ten if you switch CLM off, one local provider resident per container by default, released after ten idle minutes. Put a DECIDEALOT_TYPESAFE_API_KEY in .env and it also serves the real hosted Jev, under whatever names your TypeSafe account lists, and that’s the one route where your state does leave the box, straight to TypeSafe’s API. POST /decidealot/v1/systemone for REST, POST /decidealot/v1/systemone/batch for a pile of independent requests in one go, and four MCP tools, system_one, system_one_batch, list_models and unload_models, at /decidealot/mcp that also join the aggregated /mcp/.
CPU at /decidealot/, CUDA at /decidealot-cuda/, side by side if you want both, sharing one ~5.3 GB model directory so the weights land on disk once. DECIDEALOT=1 / DECIDEALOT_CUDA=1. CLM is on by default now, and flipping either decidealot flag also starts llama.cpp CUDA for it, which means you need an NVIDIA GPU. CPU-only box? Set DECIDEALOT_CLM_ENABLED=false before you start it. It’s a 75 MB projection head that can’t read a single word on its own, so it asks the stack’s own Qwen3 8B embeddings on llama.cpp CUDA, over the internal network, and the thing being judged still never leaves your box. It decides, your code acts. The whole story is in the decidealot post.
audiolla — Audio Production
This one’s absurd in scope. Stem separation (Demucs / UVR). Restoration — de-reverb, de-echo, de-noise. Mastering via matchering and pedalboard chains, with curated presets like master-for-spotify, podcast-cleanup, vocal-cleanup. MIR analysis: BPM, key, LUFS, beats, onsets, melody, chords, segments. DSP transforms through sox and ffmpeg. Loudness normalization. Speech enhancement (DeepFilterNet). VAD (silero). Diarization (pyannote). CLAP embeddings and zero-shot audio classification. AudioSet tagging. Audio→MIDI via basic-pitch, and MIDI compose / inspect / transform / render through fluidsynth.
Plus text-to-audio generation — stable-audio-open, musicgen, riffusion, audioldm2 — CUDA only.
Async jobs and webhooks throughout. The v1.0+ contract is strict: JSON body on every audio endpoint, raw bytes only at PUT /v1/files/{path}, and output_path xor output_url mandatory on anything that produces audio. Staged uploads and outputs get deleted 24 hours after they were last written, so download what you want to keep, or set AUDIOLLA_FILES_TTL=0 and they stay forever.
flickies — Video and Lipsync
Lipsync via LatentSync 1.5 (ByteDance, Apache-2.0, ~8 GB VRAM, the default on CUDA) and Wav2Lip / Wav2Lip-GAN (LRS2, non-commercial, gated). Face restore via GFPGAN v1.4. ffmpeg ops — trim, concat, transcode including gif/fps/codec changes, scale, mux audio, extract audio, thumbnail grid. ffprobe info. Async jobs and webhooks. 11 MCP tools, same JSON-body and output_path xor output_url contract as audiolla.
GFPGAN and LatentSync 1.5 are CUDA-only. The CPU image still does ffmpeg ops and a very slow Wav2Lip.
Local Inference — Slower, But Yours
Five local engines. Every one of them has a CPU mode.
This is the “poor people’s computer” part. Your old laptop with 16GB of RAM can run text generation, image generation, transcription, and speech synthesis. No API key. No network. No rate limit. No bill. It’s slower than cloud. Sometimes much slower. But it’s yours, it’s private, and it works offline.
Ollama — llama3.2:3b, qwen3:4b, smollm2:1.7b, qwen2.5-coder:1.5b, qwen2.5-coder:3b, phi4-mini (reasoning, 128K context), gemma4:e2b (vision), gemma3:4b (vision), nuextract-v1.5 (structured text → JSON extraction), dolphin-phi, plus two embedding models for RAG. The smallest needs 1GB of RAM. OLLAMA_CUDA=1 adds the heavier set: qwen3:8b, qwen3:30b-a3b (MoE), qwen2.5vl:7b (vision), gemma4:e4b (vision), deepseek-coder-v2:16b (MoE, 160K context), deepseek-r1:8b and deepseek-r1:14b (reasoning), phi4-reasoning:plus (reasoning), qwen3-abliterated:16b (uncensored), gemma4-abliterated:e4b (uncensored vision), qwen2.5-coder:7b, llama3.1:8b. Flash attention, quantized KV cache, shared model storage with the CPU service — no duplicate downloads.
talkies — the unified ASR + TTS service, and the replacement for what used to be Speaches in this stack. CPU now ships 13 models: whisper-large-v3 and -turbo, canary-180m-flash, nemotron-3.5-asr-0.6b, four Sherpa-ONNX Zipformer variants, Vosk small English, and two phoneme recognisers that hand you a raw IPA phone stream with per-phone timestamps instead of words; Kokoro for TTS. CUDA is up to 22, adding parakeet-tdt-0.6b-v3, canary-1b-flash, canary-qwen-2.5b, Chatterbox Turbo, and the Qwen3-TTS family — 0.6B and 1.7B, with voice cloning, voice design, and emotion control through the instructions field. Chatterbox is the only one of them that watermarks its output at all: ResembleAI’s PerTh neural watermark, on by default, and TALKIES_CHATTERBOX_WATERMARK=false turns it off for clean audio, while Kokoro and Qwen3-TTS never embed anything. OpenAI-compatible endpoints, so existing Whisper calls work unchanged. TALKIES=1 / TALKIES_CUDA=1.
The Sherpa and Vosk additions land as ten new transcription aliases — the five models under local-talkies-, then the same five again under local-talkies-cuda-. The int8 builds are quantized (smaller and faster, slightly lower accuracy); left-64 versus left-128 sets the attention left-context window. All five do native live ASR over the streaming WebSocket as well as ordinary POST /v1/audio/transcriptions, and the CUDA image installs a hash-verified upstream Sherpa CUDA wheel so the four Sherpa ones actually use its CUDA execution provider instead of quietly falling back to CPU inside a GPU container. Vosk decodes on CPU either way.
One fallback subtlety worth stealing: six chains in fallbacks.json used to list a sibling talkies model first — whisper-large-v3 → whisper-large-v3-turbo, and the equivalents for the CUDA variant and both TTS entries. That can never work. Each talkies container serves one model at a time and evicts the previous one on admission, so a sibling hop doesn’t find spare capacity, it forces a model swap — and under retries that’s just repeated eviction churn. The chains route to the other talkies container first now, then out to the cloud providers. CPU↔CUDA hops stay, because those genuinely are independent processes.
stable-diffusion.cpp — local image generation. CPU runs sd-turbo and sdxl-turbo out of the box. SDCPP_CUDA=1 for hardware acceleration and the full model set: sd-turbo, sdxl-turbo, sdxl-lightning, flux-schnell, juggernaut-xi. A pull sidecar downloads the weights once, before the server first starts, and caches them locally. The OpenAI-compatible /images/generations endpoint means existing code works as-is.
vLLM — local text LLMs plus embeddings. qwen3-0.6b for generation on both. Embeddings are nomic-embed-v2 on CUDA, while CPU gets bge-m3 (multilingual) and nomic-embed-v1.5, because Nomic v2 is a MoE model, the vLLM CPU build has no MoE kernels, and the old CPU alias failed every single request. Indexed anything with it? Embed it again, vectors from different models don’t mix. VLLM=1 / VLLM_CUDA=1.
llama.cpp — a GGUF wrapper with two models now. Surya OCR 2 does document understanding, on CPU and CUDA. CUDA also gets local-llamacpp-cuda-qwen3-8b, Qwen3 8B at Q8_0 serving embeddings only: 4096 dimensions, last-token pooling, L2-normalized, and it’s the encoder decidealot’s CLM thinks through. Don’t send it chat, it doesn’t do chat. Every download is pinned to an exact revision and SHA-256, and the CPU and CUDA profiles each get their own pull sidecar that fetches only its own files, so a CPU-only box never pulls the 8.7 GB Qwen, unless decidealot is on there with CLM left at its default, because that starts the CUDA profile anyway.LLAMACPP=1 / LLAMACPP_CUDA=1.
Local models sit in the speech, image, embedding and coding fallback chains. Cloud fails there? Local picks it up. Most general chat chains end in the cloud, so for those you target local directly: "model": "local-ollama-cpu-llama3.2-3b". Zero network. Zero cost. Slower, but it answers.
One GPU, Everything
Here’s the engineering problem: you have one GPU. Ollama wants VRAM for the LLM. sd.cpp wants VRAM for image generation. talkies wants VRAM for transcription and speech. audiolla wants it for stem separation. flickies wants ~8GB of it for LatentSync. Load them all and you OOM.
The resource manager solves this automatically. A LiteLLM callback enforces mutual exclusion per hardware — one CUDA job at a time. The lock lives in Redis, so it holds across every LiteLLM worker instead of each worker trusting its own private semaphore and four of them piling onto one GPU. Redis down? The request fails. It does not sneak through unlocked. When an image generation request arrives while an LLM is loaded, the resource manager takes the lock, unloads the LLM, then lets the image generation proceed. When a TTS request comes after that, it unloads the image generator first. Same logic on CPU.
Every service has its own unload API and the resource manager knows all of them: Ollama takes keep_alive: 0, sd.cpp has POST /sdcpp/v1/unload, talkies / vllm-cuda / llamacpp-cuda take DELETE /api/ps/{model_id} per model or POST /unload to kill whatever’s loaded, audiolla has POST /v1/unload to bulk-evict every loaded engine, and flickies has no bulk call, so the manager lists its engines with GET /v1/engines and kills the loaded ones one by one with DELETE /v1/engines/{slug}. predictalot and decidealot both take POST /v1/models/unload and answer 409 when they’re mid-request, in which case the manager logs it, moves on, and lets their own idle timer free the model later.
There are operator-facing endpoints too, when you want to nuke it all by hand: POST /v1/unload/cuda fans out concurrently to ollama-cuda, sdcpp-cuda, talkies-cuda, vllm-cuda, llamacpp-cuda, audiolla-cuda, flickies-cuda, predictalot-cuda and decidealot-cuda. POST /v1/unload/cpu does the same nine on the CPU side. POST /v1/unload runs both in sequence and hands you back a per-service report of what actually got evicted, and a service that’s busy serving a request reports "status": "busy" and keeps its model. predictalot inference tools called through the aggregated /mcp/ take the same lock as everything else, and decidealot now takes it itself for Laya and Von on every route: direct REST, MCP and every single batch item. The hole that’s left: audiolla and flickies never take the lock at all, and predictalot hit directly on its own route skips it too. None of those evict anything on the way in, and decidealot only kicks out its own llama.cpp encoder before a CUDA load, so on a small GPU a cold start there can still OOM next to a loaded Ollama model.
You don’t manage any of this. You send requests. The platform juggles VRAM automatically. The only cost is latency — the first request after a swap includes model load time. After that, it’s fast until the idle timeout unloads the model to free memory for the next thing.
Web Search
SearXNG at /searxng/. Self-hosted meta-search — queries Google, Bing, DuckDuckGo, and Wikipedia simultaneously. No API key. Runs entirely locally.
Fix your secret_key if you deployed this early. SearXNG’s was hardcoded in the tracked searxng/settings.yml. It’s generated per deployment now and read from SEARXNG_SECRET_KEY in .env — set it with openssl rand -hex 32 when you flip SEARXNG=1, or SearXNG comes up on the upstream placeholder. The tracked settings.yml is gone entirely; settings render from a searxng_config entry in the configs: block, the same way proxq_config already sources ${REDIS_PASSWORD}. Exposure was limited — /searxng/ sits behind nginx admin auth, the container publishes no host ports, and limiter: false meant the key wasn’t gating rate limits — but the old value is still sitting in git history, which is the part that doesn’t un-happen.
The MCP search_web tool means any function-calling model, driven by an MCP-capable client, can search the web on its own. The model decides it needs to look something up, calls the tool, gets results, continues reasoning. You didn’t build a search integration. You flipped SEARXNG=1.
LibreChat — The Daily Driver
Everything above works from any MCP-capable client. But for daily use, there’s LibreChat at /librechat/.
All models in the dropdown. All MCP tools wired in. Pick a model, start talking. The model can still invoke the browser, storage, the coding agents, image generation, TTS, and web search autonomously — LibreChat runs the tool loop for you, no client code needed. Conversation history backed by MongoDB. File uploads. WebSocket streaming.
First registered user becomes admin. Set LIBRECHAT_ALLOW_REGISTRATION=false after that and you’re the only one in.
Enable it: LIBRECHAT=1 in .env.
Async Queue
Long inference requests time out. Hit /q/ instead of / and the request goes into a Redis-backed queue. You get a job ID back instantly. The actual inference runs in the background. Poll for status, fetch the result when it’s done.
# submit — returns 202 immediately
curl http://localhost:4000/q/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"model": "cerebras-gpt-oss-120b", "messages": [{"role":"user","content":"write a novel"}]}'
# → {"jobId": "550e8400-e29b-41d4-a716-446655440000"}
# check status
curl http://localhost:4000/q/__jobs/550e8400-e29b-41d4-a716-446655440000 \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
# get the result
curl http://localhost:4000/q/__jobs/550e8400-e29b-41d4-a716-446655440000/content \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"Configurable concurrency, retention, timeouts, retries, and response caching. Only OpenAI API paths get queued — health checks and admin requests pass through directly. LiteLLM has its own response cache on top of that, also in Redis: send the exact same request twice inside 10 minutes and the second one comes straight from the cache without touching the model. Great for your bill, shit if you’re resampling one prompt hoping for a different answer.
It Ships Its Own Agent Skill
The repo carries .agents/skills/aigate/ — a ClawHub-published agent skill that teaches a coding agent how to drive the stack. So instead of explaining your own infrastructure to an assistant every session, you install the skill and it already knows the endpoints, the flags, and the tool contracts.
Security
Internal services — PostgreSQL, MongoDB, Redis, the browser cluster, the storage backend — have no host ports. They’re on isolated Docker networks. Nothing reaches them from outside the stack. The only exposed surface is nginx on port 4000, and that requires bearer token auth. Redis doesn’t even have a default user any more: proxq and LiteLLM each log in as their own ACL user that can only touch their own keys. Got something of yours talking to aigate’s Redis with just REDIS_PASSWORD? It needs the username proxq now, and the password can’t contain whitespace.
Application containers run with no-new-privileges:true. Two carve-outs, both deliberate and both documented in the compose file: piston needs privileged: true because nsjail has to build user namespaces and chroots per execution — the actual isolation lives inside the nsjail subprocess, and the privileged container is what nsjail needs in order to construct that jail. tailscale needs NET_ADMIN and NET_RAW for its kernel tun device. Everything else drops privileges. make run validates that any file paths in .env actually exist before starting — no silently broken volume mounts.
Want it publicly reachable without opening a firewall port? CLOUDFLARED=1 — Cloudflare Tunnel. DDoS protection, TLS termination, no open ports, no IP to scan. Quick tunnel for a random *.trycloudflare.com URL, or a named tunnel for a fixed domain.
Don’t want it public at all, but still want to reach it from your laptop, phone, or another box? TAILSCALE=1 with TS_AUTHKEY=tskey-auth-... and a TS_HOSTNAME. A Tailscale sidecar joins your tailnet and runs tailscale serve as an HTTP reverse proxy straight to nginx-edge:4000, nginx’s alias on a dedicated ingress network. That’s the only door where nginx trusts the forwarded client IP, so it sees who’s actually calling instead of the sidecar’s address, and MCP can tell tailnet callers from public ones without believing a header some client made up. No HTTPS auto-cert (TLS termination, if you want it, lives in nginx). Access via http://<TS_HOSTNAME>.<tailnet>.ts.net — plain HTTP is fine here because WireGuard already encrypts every byte inside the tailnet. Works with hosted Tailscale or self-hosted Headscale (use TS_EXTRA_ARGS=--login-server=..., plus TS_SERVE_FQDN set to the node’s real DNS name, which a hosted tailnet without a certificate domain needs too). State persists in .data/tailscale/ so reboots reuse the existing login. Combine it with the bearer token auth and you’ve got two completely independent layers of access control.
That’s traffic coming in. Traffic going out is a separate thing you can now turn on: tailnet egress for the coding agents, so claudebox, pibox-zai and pibox can all reach machines on your tailnet instead of only the public internet. It ships as a compose overlay (docker-compose.tailscale.yml) and needs two more values, TS_MAGICDNS_SUFFIX (your tailnet’s MagicDNS suffix, straight out of tailscale status) and TS_FALLBACK_DNS (default 1.1.1.1). The second one exists because DNS ends up split: MagicDNS answers tailnet names and hands back SERVFAIL for everything else, so public names need a resolver to fall through to.
Hundreds of tests. Health checks, routing, auth, MCP tool validation, storage CRUD, browser automation, agentic coding runs, async job lifecycle, local TTS/STT round-trips, CUDA resource manager verification, local image generation, LLM-to-MCP end-to-end tool calling, SearXNG search, Telethon MTProto round-trips against a real account. Plus security: cross-token isolation, session hijack attempts, HTTP request smuggling (CL.TE/TE.CL), h2c smuggling, SSRF via browser and MCP to internal services, prompt injection key extraction, path traversal, S3 presign abuse, stored XSS, model name injection, header injection, Docker socket isolation. This isn’t a hobby project test suite. This is paranoia as a feature.
Setup
git clone https://github.com/psyb0t/aigate && cd aigate
make bootstrap # seeds .env; then edit it: add keys, flip flags
make run-bgEvery variable is documented in .env.example. Enable what you have, ignore what you don’t. You don’t need to copy that file by hand any more either: make bootstrap seeds .env, and every other target seeds it first, so make run on a fresh clone just works.
When you want to change something about the stack itself rather than a flag, put it in docker-compose.override.yml. That’s the supported seam now: the Makefile assembles COMPOSE_FILE out of the base file, the tailscale overlay when you’ve enabled it, and your override last. Edit the base compose file directly and the next update eats your changes; the override survives.
If resources matter — and on a normal computer they do — every service has a fixed memory and CPU limit, the default in the compose file, and you override one in .env (TALKIES_CUDA_MEM_LIMIT=16g). Nothing scales with your RAM, because a model needs the same memory on any box and a limit below its load size just gets the container OOM-killed. make limits prints every enabled service with its limit, warns you when the worst case doesn’t fit, and writes CPU caps to .env.limits only where a default asks for more cores than you’ve got. CUDA services are resource-manager-aware, so the budget counts the largest one — not every GPU service at full allocation simultaneously. Upgrading from an older aigate? Run make limits again, because an old .env.limits still holds percentage-based memory limits that override the new defaults and can starve talkies-cuda below what its models need to even load.
# free tier, auto-fallback
curl http://localhost:4000/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"model": "groq-gpt-oss-120b", "messages": [{"role":"user","content":"hello"}]}'
# local, no network, no limits
curl http://localhost:4000/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"model": "local-ollama-cpu-llama3.2-3b", "messages": [{"role":"user","content":"hello"}]}'
# image generation (FLUX, local CUDA)
curl http://localhost:4000/images/generations \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"model": "local-sdcpp-cuda-flux-schnell", "prompt": "a cat riding a skateboard"}'
# local image generation (no network, no cost)
curl http://localhost:4000/images/generations \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"model": "local-sdcpp-cpu-sd-turbo", "prompt": "a red panda in a forest", "size": "512x512"}'
# transcription (local, CPU)
curl http://localhost:4000/audio/transcriptions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-F "model=local-talkies-whisper-large-v3-turbo" -F "[email protected]"
# text-to-speech (local, multiple voices)
curl http://localhost:4000/audio/speech \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"model": "local-talkies-kokoro-tts", "input": "Hello world", "voice": "af_heart"}' \
-o speech.mp3
# plain chat call, no tools attached (tools need an MCP client on /mcp/)
curl http://localhost:4000/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"model": "groq-qwen3.8-27b", "messages": [{"role":"user","content":"explain rust ownership in two sentences"}]}'
# async — submit and poll
curl http://localhost:4000/q/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"model": "cerebras-gpt-oss-120b", "messages": [{"role":"user","content":"write a novel"}]}'Five providers with free tiers, five local engines, the rest as fallback. Slower on a normal computer — but it runs, it’s private, and nobody can rate-limit you out of your own infrastructure.
github.com/psyb0t/aigate
Installing It Into Your Agent
That skill isn’t OpenClaw-only any more. Everything under .agents/ is catalogued in one marketplace, so it is two commands:
claude plugin marketplace add psyb0t/agents
claude plugin install aigate@psyb0tCodex uses the same marketplace with a different verb — codex plugin add aigate@psyb0t, because there is no codex plugin install. It also finds the skill on its own in a checkout of the repo, since it scans .agents/skills/ natively with nothing installed at all.