Local LLM Demo

CADS-DEMO-local-llm (GitHub) exposes a local-GPU model — qwen3-coder:30b, served by Ollama on a single NVIDIA TITAN RTX (24GB VRAM) — to the internet through a real CADS-Tunnel agent, fronted by LiteLLM for per-person API keys and an OpenAI/Anthropic-compatible surface.

The live endpoint is llm-34a13a96.bunsenbrenner.org — a real, running proxy to real local GPU inference, not a mockup. Every example response in this documentation is copied from an actual call against it.

measured Checked live 2026-08-29: GET /health/liveliness returns "I'm alive!" with no auth, and the proxy refuses an unscoped model on a scoped key. The model responses quoted throughout are audited — real runs the maintainer recorded, quoted from transcripts rather than re-issued on every read.

Start here

• Call the API and point Claude Code at it — get a scoped key, make a real request, then point an actual coding-agent CLI at the same endpoint and see how its behavior differs from a plain API call.

Sections

• Tutorials — learn by doing, start to finish.
• How-to guides — accomplish a specific task.
• Reference — look up an exact fact (models, endpoints, env vars).
• Explanation — understand why this is built this way.

How to read the provenance marks

Claims about the live system carry a mark for how they were checked:

• measured — reproduced end to end against the running proxy and quoted verbatim.
• audited — a real run the maintainer captured, quoted from stored transcripts (capability-probe/, cli-tools/) rather than re-run on every read.
• not built — deliberately absent: reserved, refused, or switched off on purpose, and documented as such.