oMNI Chat

Chapter 1

Introduction

Omni Chat is a lightweight chat application built around a single Go core. There is no database server to install and no external service to manage — conversations, projects, documents, and memory all live in a local SQLite file. The core talks to any endpoint that speaks the OpenAI API (/v1/chat/completions, /v1/embeddings, /v1/models), so it works with local engines like llama.cpp and vLLM, a LiteLLM proxy, or hosted providers.

Two shells, one core

The same Go core powers two front ends:

• Web app — runs in any browser over HTTP. This is the supported way to run Omni Chat today.
• Desktop app Planned — a native Tauri shell that bundles the core as a sidecar. The shell has not shipped yet; use the web app.

Chapter 2

Getting Started

The fastest way to run Omni Chat is Docker. This chapter walks through everything end-to-end: getting an AI backend running if you don't have one yet, picking how much of the optional stack (web search, URL fetch, the auxiliary model, smart routing) fits your hardware, and starting the app. Building from source instead? Skip to Local Development Setup.

What you need

Docker (Docker Desktop, or Docker Engine + the Compose plugin on Linux) and access to an OpenAI-compatible AI backend — something that answers /v1/chat/completions. That can be:

• Something you already have running (llama.cpp, vLLM, a LiteLLM proxy, an existing hosted provider) — skip ahead to Running Omni Chat.
• A model you set up locally right now with Ollama or LM Studio — see below.
• A hosted/cloud endpoint with no local model at all — see Or use a hosted endpoint instead.

Setting up a local AI backend

ollama pull llama3.2

# Docker
CHAT_ENDPOINT_URL=http://host.docker.internal:11434

# Host-run (make dev)
CHAT_ENDPOINT_URL=http://localhost:11434 make dev

# Docker
CHAT_ENDPOINT_URL=http://host.docker.internal:1234

# Host-run (make dev)
CHAT_ENDPOINT_URL=http://localhost:1234 make dev

Either way, the model you pulled/loaded shows up in Omni Chat's Model dropdown automatically — it's fetched live from the backend's /v1/models.

Or use a hosted endpoint instead

No GPU, or you'd rather not run a local model at all: point Omni Chat at a cloud endpoint. This is also the recommended path at 8GB of memory or less — see How much memory do you have? below.

• Ollama Cloud — an OpenAI-compatible hosted endpoint with no local install. Create a key at ollama.com/settings/keys (a free tier exists), then:

  CHAT_ENDPOINT_URL=https://ollama.com
  CHAT_API_KEY=<your ollama.com key>

  Pick from cloud-hosted models like gpt-oss:120b, qwen3-coder:480b, or deepseek-v3.2 in the Model dropdown — no local GPU or RAM spent on inference.
• RouteLLM — an open-source router (lm-sys/RouteLLM) that fronts a cheap/fast model and a strong/expensive one and exposes its own OpenAI-compatible endpoint, routing each request to control cost. Self-host it and point CHAT_ENDPOINT_URL at wherever it listens (hosted RouteLLM-compatible options also exist). This is a different routing axis than Omni Chat's own Auto (Smart Router) — RouteLLM chooses between two backends for cost; Auto chooses among your configured backends for task fit — and the two can be combined.
• Any other hosted OpenAI-compatible provider you already use — just set CHAT_ENDPOINT_URL and CHAT_API_KEY.

How much memory do you have?

Search and Fetch are lightweight: SearXNG is CPU/RAM only, and the Fetch reader sidecar is a headless-browser process — neither touches VRAM or unified memory, so they're safe to add regardless of hardware. The chat model itself, plus the optional Auxiliary model (recommended: LiquidAI/LFM2.5-1.2B-Instruct, ~730MB GGUF), the Router classifier (recommended: katanemo/Arch-Router-1.5B, ~1GB GGUF), and the Image generation sidecar (see the note below the table) all compete for the same VRAM/unified-memory pool — size your setup by how many of those you plan to run at once.

Running Omni Chat: pick your combination

Each step below just adds one more -f flag to the previous command — start barebones and layer on whichever of these your memory tier and needs call for. What each overlay does in depth (healthchecks, image pinning, reader vs. direct fetch, running any sidecar without Compose) is covered in Administration & Deployment; this is just the fast path.

# Once, before any of the combinations below
cp .env.example .env

a. Barebones — chat only, no web search:

docker compose up -d

b. + Search — adds self-hosted web search (SearXNG):

printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" >> .env
docker compose -f compose.yaml -f compose.search.yaml up -d

c. + Fetch — adds JS-page rendering and YouTube-transcript support for the fetch_url tool:

docker compose -f compose.yaml -f compose.search.yaml -f compose.fetch.yaml up -d

d. + Auxiliary model — adds a small dedicated model for chat titles, memory jobs, and (with no dedicated classifier configured) smart-router classification:

docker compose -f compose.yaml -f compose.search.yaml -f compose.fetch.yaml \
  -f compose.aux.yaml up -d

e. + Auxiliary model + Router classifier — the full stack; adds a dedicated routing model that outranks the auxiliary model for smart-router classification:

docker compose -f compose.yaml -f compose.search.yaml -f compose.fetch.yaml \
  -f compose.aux.yaml -f compose.router.yaml up -d

f. + Agent handoff & SFTP — lets the assistant delegate real coding tasks to a containerized agent (each conversation gets a private workspace) and gives users SFTP access to their files. The core runs each approved delegation as a sibling container through the host's Docker socket, so this overlay needs a one-time setup before the first run.

1. Build the agent image. This is the container each delegation runs in (Pi plus git):

docker build -t omni-agent-pi docker/agent-pi

2. Create the shared workspace directory. Managed workspaces live here, and the same absolute path must exist on the host, in .env, and in agents.json — because the host Docker daemon resolves the sibling container's mounts against host paths. On Linux:

sudo mkdir -p /srv/omni-agent-work
sudo chown 10001:10001 /srv/omni-agent-work   # 10001 is the core container's user

On macOS (Docker Desktop) the /srv path is on the sealed read-only system volume, so put the directory under your home instead — and skip the chown, since Docker Desktop maps ownership automatically:

mkdir -p ~/omni-agent-work

3. Add the agent config. Copy the example and set managed_root to the directory from step 2 (a literal absolute path — no ~, which the container cannot expand):

cp docs/agents.compose-example.json config/agents.json

The example is managed-mode with a Docker-runtime Pi agent and SFTP enabled. On macOS, edit managed_root to e.g. /Users/you/omni-agent-work.

4. Pre-generate the SFTP host key. The container mounts /config read-only, so the server cannot create its key on first run — make it ahead of time (no chown needed on macOS):

ssh-keygen -t ed25519 -f config/sftp_host_key -N ""
sudo chown 10001 config/sftp_host_key          # Linux only

5. Point .env at the workspace — this value must match managed_root exactly:

echo 'OMNI_AGENT_WORK_DIR=/srv/omni-agent-work' >> .env    # macOS: /Users/you/omni-agent-work

6. Bring the stack up with the overlay (stackable with the others). On Linux, pass the group that owns the Docker socket so the non-root core may use it; on macOS Docker Desktop, omit DOCKER_GID entirely:

DOCKER_GID=$(getent group docker | cut -d: -f3) \
  docker compose -f compose.yaml -f compose.agents.yaml up --build -d

docker compose -f compose.yaml -f compose.agents.yaml up --build -d   # macOS Docker Desktop

g. + Image generation — lets the assistant create images ("generate an image of a cyberpunk cat riding a skateboard") with a bundled Z-Image-Turbo sidecar. It requires an NVIDIA GPU (with the NVIDIA Container Toolkit installed; ~16 GB of VRAM) and downloads ~12 GB of model weights on first start:

docker compose -f compose.yaml -f compose.imagegen.yaml up --build -d

Watch docker compose logs -f imagegen on the first run — the app waits for the sidecar's healthcheck, which stays starting until the weights are downloaded and loaded. The default model is Tongyi-MAI/Z-Image-Turbo; image edits fall back to img2img re-imagining with those same weights unless you set IMAGEGEN_EDIT_MODEL_ID in .env to a dedicated instruction-edit model — see the verified choices and their (large) sizes in .env.example and the memory note above. No NVIDIA GPU on this machine? Run compose.imagegen-only.yaml on a GPU box and set CHAT_IMAGE_GEN_URL=http://gpu-host:8001 in .env instead — or point it at any other server that speaks the OpenAI Images API. See Generating images for how it behaves in chat.

Jetson/L4T or arm64 CUDA hosts: the default base image is x86-64 CUDA, so override IMAGEGEN_BASE_IMAGE in .env with a CUDA-enabled PyTorch image matching your host's driver stack — torch is not reinstalled by the build, so it must come from the base — and set IMAGEGEN_RUNTIME=nvidia to expose the GPU. For example IMAGEGEN_BASE_IMAGE=dustynv/pytorch:2.7-r36.4.0 on a Jetson (match your JetPack), or nvcr.io/nvidia/pytorch:25.01-py3 for arm64 CUDA (DGX Spark / Grace-Hopper).

Whichever combination you run, open http://127.0.0.1:8080. The default AI endpoint is http://host.docker.internal:8000; edit CHAT_ENDPOINT_URL in .env to point at the backend you set up above. See Chapter 12 for running without Compose, pinning an image tag, backups, and upgrades.

First run: create the owner

The first time you open a fresh instance, no users exist yet. You'll be taken to a setup screen to create the owner account (username, display name, password). After that, the instance shows a login screen on every visit.

Add to iPhone Home Screen

When the web app is reachable from your phone, open it in Safari, tap Share, then tap Add to Home Screen. iOS uses the bundled Omni app icon and launches Omni Chat in a standalone browser window from that Home Screen icon.

Building from source instead of using Docker? See Local Development Setup. Ready to combine multiple backends or try the Auto Smart Router? See Models & Backends.

Chapter 3

Local Development Setup

For contributors, or anyone who'd rather build from source than use Docker. If you just want to run Omni Chat, see Getting Started instead.

Prerequisites

Building Omni Chat from source needs the following toolchain. Everything runs through the Makefile, so make is required too.

To actually run the app you also need an OpenAI-compatible endpoint already listening — see Getting Started for setting one up with Ollama, LM Studio, or a hosted provider if you don't have one. That's a runtime dependency, not a build dependency.

The README has copy-paste install commands for macOS (Homebrew) and Ubuntu/Debian.

Quick launch

Point the app at a running endpoint and start the development build. This compiles the web app and runs the core with the SPA embedded:

# Ollama defaults to port 11434, LM Studio to 1234, llama.cpp/vLLM commonly to 8000/8080
CHAT_ENDPOINT_URL=http://localhost:11434 make dev

make dev serves the app on port 8080, so open http://127.0.0.1:8080 in your browser. The core also prints one handshake line to stdout when it's ready:

READY port=8080

Add CHAT_PORT to use a different port, CHAT_API_KEY if your endpoint needs a token, and CHAT_DB_PATH to choose where the SQLite file lives — see Chapter 11 for the full list. Running the compiled binary directly (Chapter 12) defaults CHAT_PORT to 0, which picks an ephemeral port and reports it in the READY line. First run creates the owner account the same way as the Docker path — see Getting Started.

Running modes during development

See Chapter 12 for production builds and running the compiled binary.

Chapter 4

The Interface

Omni Chat uses a three-column, adaptive layout. The side columns collapse to give the conversation more room; the right Settings panel starts collapsed by default, and each panel remembers its expand/collapse state per user.

The top-right user button opens the account menu. From there you can edit your username, display name, and local avatar image, toggle Memory for the current chat, manage saved memories, or sign off.

The composer row also carries lean Model and Profile dropdowns so you can switch either without opening the Settings panel. The rest of this manual walks through each control.

Light & dark theme

The top bar carries a sun/moon button beside the switch-user button that toggles between the light and dark themes. The theme defaults to your operating system's preference and is remembered per user in the browser, so each account keeps its own choice on that device.

Chapter 5

Chatting

Starting and organizing chats

• New chat — start a conversation inside a project or leave it unassigned. After the first answer completes, Omni Chat may replace the initial first-message title with a short generated summary of that first exchange.
• Projects — group related chats. A project can hold shared instructions that are added to every chat inside it (see Chapter 8).
• Hashtags — type # to create or attach a tag (for example #research or #implementation). Tags appear as chips and filter the left sidebar.

Chat Details

The Chat Details control in the top bar lets you rename a chat, move it to another project, add or remove hashtags, archive it, or delete it.

Sharing & export

The share icon in the top bar exports the current conversation as a standalone document. Both options render what's already on screen into one self-contained HTML page — no server round-trip.

• Download HTML saves a single .html file with the chat title, every message, rendered markdown, code-run outputs, and web-source citations. Image attachments are inlined as data URLs so the page opens fully offline; other attachments are listed by filename. Assistant thinking is not included.
• Save as PDF opens that same document in a print view — choose your browser's Save as PDF destination to keep crisp, selectable text without any extra tooling. (Allow pop-ups for the app if the print view doesn't open.)
• Download Markdown saves a .md file with each assistant answer kept as Markdown, code-run outputs as fenced blocks, and web sources as links. Image attachments are referenced by filename rather than embedded, so the file stays small and portable.

Working with messages

• Edit & resend — change one of your earlier messages and re-run from that point.
• Edit response — revise an assistant response from a short edit instruction, such as adding sound to a generated game or rephrasing one paragraph. The original stays visible while the edit runs; revision thinking can be expanded, and the answer is replaced only after the edit succeeds. The instruction is not saved as a chat message.
• Regenerate — re-run the assistant's answer from any prior turn (handy after switching model or profile).
• Delete with undo — remove a message with a short undo window.
• Stop — cancel a streaming response; the partial answer is kept.
• Generation details — hover or focus the info icon for a quick view, or click it to pin the panel until you close it, click elsewhere, or press Escape.

Writing code in the composer

Type three backticks on an otherwise empty composer line to open a monospace code block. Paste or type code there without syntax highlighting; Enter and Shift+Enter both add another line and never submit. Press Arrow Down at the end of the block, or click the blank line below it, to continue with normal text. A populated code block disappears only after all of its contents are deleted or cut; Backspace dismisses a newly opened empty block. After sending, the user message keeps the visual code block while ordinary user text remains literal.

The Generation details panel shows the profile used (or No profile), the exact model and endpoint that handled the response, thinking mode, and the tool outcomes for the response — web access, and, when those features were in play, code execution and agent delegation — plus labeled input/output counts, prefill and generation speed, time to first token, thinking time when available, and total request-to-completion time. Omni Chat saves these details with each new assistant response, so changing a chat's settings later does not rewrite its history. Older responses show only values that can be proved from their saved data and label the rest Not recorded.

Attaching files (images & documents)

Use the paperclip button in the composer — or drag files onto it, or paste an image from the clipboard — to attach files to your next message. Supported types: images (PNG, JPEG, WebP, GIF) and documents (PDF, plain text, Markdown, HTML), up to 20 MB per file and 8 files per message. Attached files appear as chips; remove one with its ✕ before sending. Sent images show as thumbnails in the chat history and documents as clickable chips.

• Images are sent to the model as image content on that turn, so you can ask "what's in this photo?". This needs a vision-capable model (for example a local Qwen2.5-VL served by llama.cpp or Ollama); the composer warns you when the selected model cannot view images, and Auto (Smart Router) picks a vision model automatically. Large images are downscaled server-side so local inference stays fast. To keep follow-up turns cheap on local hardware, images are only sent in full on the turn you attach them — later turns reference them by name.
  Vision support is detected from, strongest first: model_overrides in endpoints.json; the endpoint's own reported modalities (the llama.cpp router exposes input_modalities per model — a llama-server started without --mmproj reports text-only and is treated as such, so load the mmproj file to enable images); and a built-in catalog of known multimodal families (Gemma 3/4, Qwen 3.5/3.6, Qwen-VL, LLaVA, Pixtral, GPT-4o/5, Claude, Gemini). If a vision model is still misdetected, add {"model_pattern": "…", "vision": true} to that endpoint's model_overrides.
• Documents have their text extracted and given to the model as source context, on the turn you attach them and on follow-up questions in the same chat (within the model's context budget).
• Scanned PDFs with no text layer are handled by extracting their page images and sending those through the vision path — chat about a scanned document exactly like a photo. If nothing can be extracted, the upload is rejected with a clear message.

Reasoning / thinking traces

When a model emits a separate thinking or reasoning block, Omni Chat keeps it apart from the answer and shows it in a collapsible region. Reasoning is stored separately and is never fed back into later prompts — only the answer content is. The compact brain switch beside the composer Profile selector and the labeled switch in Settings are synchronized and update the same per-chat state. When the leading reasoning phase ends, its folded title shows the elapsed time from the first reasoning token to the first visible answer token; this duration remains available after reloading the chat. If the selected model has no known reasoning channel or adapter, the Thinking switch is shown as unavailable. OMLX models are detected from their chat-template capability metadata rather than a model-name allowlist.

Code preview

Assistant code blocks include toolbar controls to copy or save the displayed code. The save action opens a Save As picker when the browser supports it, otherwise it falls back to a normal local download. Named blocks keep their filename and unnamed blocks use a language-based snippet.* name; both include an ordering suffix such as main-01.py or snippet-02.js. HTML code blocks in an assistant message can be rendered in a sandboxed preview that runs inline CSS and JavaScript, including dynamically evaluated code used by apps such as calculators, and may load resources over HTTPS from a CDN. Adjacent css and js blocks in the same message are bundled into the same preview. Use the popout control on an HTML preview to open the same sandbox in a new browser tab/window; the inline message returns to code view while keyboard-driven demos such as games keep focus in the popout.

Running code blocks

JavaScript, TypeScript, Python, Go, and bash code blocks show a Run button that executes the snippet locally, in a sandboxed WebAssembly runtime (QuickJS for JS/TS, Pyodide/CPython for Python, a Yaegi interpreter for Go, WASIX GNU bash for bash/sh) inside a dedicated Web Worker — nothing is sent to a server. Output (stdout and stderr, the exit code, and how long it took) streams into a panel below the code, with Stop, Re-run, and Close controls. The runtime downloads once on first use and is then cached (the Python runtime is about 6 MB, Go about 8 MB, bash about 4 MB).

Interactive programs: a JavaScript or TypeScript program that reads input with Node readline or process.stdin, a Python program that calls input(), or a bash block runs in a small terminal below the block — type into it and the run keeps going. For JS/TS the run pauses while it waits for a line (idle time doesn't count against the 30-second limit; a runaway loop after you answer still does) and works in any browser with no special setup. Python input() and bash need a cross-origin-isolated context (a SharedArrayBuffer): they work in Chrome and Firefox and show a "not supported" notice in Safari. bash runs your block as a real script (bash -c) with coreutils (ls, cat, and friends); if the script calls read, the terminal stays live for input. It runs without the 30-second wall-clock limit (Stop and the 1 MiB output cap are the guards). Because bash runs in WebAssembly without full POSIX signals (no SIGPIPE), a filter reading an unbounded stream piped into another command — e.g. tr < /dev/urandom | head -c 16 — produces no output; bound the source first (head -c 64 /dev/urandom | …).

Multi-file programs: when code blocks in one answer carry filenames — on the fence (```python main.py) or as a bold/heading line right above the block (**utils.py**) — they form a workspace. Running a named block mounts its named siblings into the sandbox's in-memory filesystem, so import utils, a TypeScript import './greet', or a multi-file Go package works. The files exist only for that run and are discarded with it; blocks without filenames run alone exactly as before. If the same filename appears twice in an answer, the later block wins.

Letting the model run code itself (Code Execution)

With a tool-capable model, the assistant can execute code as part of answering — for example "sort these numbers and tell me the median, use a tool". Turn on the Code Execution icon in the composer — the </> button next to Include Sources and Thinking, which outlines orange when active (off by default). The model is then offered a run_code tool for Python and JavaScript: it writes a program, your browser executes it in the same sandbox as the Run button — no network, no filesystem, no input, 30-second cap — and the printed output flows back into the model's answer. The code never leaves your machine for execution; the server only relays it to your own browser tab.

While it runs, the response shows a Running Python… chip that turns into a collapsed card such as Ran Python · ok · 0.4 s. Expand the card to see exactly what code executed and its stdout/stderr; the card is saved with the message, so you can audit it after a reload too. The model gets at most three runs per response. If the run fails or times out, the model is told so and answers as best it can — check the card when a computed answer looks off. Go and bash are not offered to the model (they stay Run-button-only), and with the toggle off nothing ever executes without your explicit click.

Handing a task to an external agent (Agent)

Code Execution runs in a sealed sandbox — no files, no network. When a task needs the real thing (edit a project, run its tests, make a git commit), the assistant can hand it to an external coding agent running on the machine that hosts Omni Chat — either installed there directly or inside a Docker container the operator configures (see Running Omni Chat, combination f, for the Compose setup). The first supported agent is Pi. This only works if whoever runs your instance has enabled it (by creating an agents.json that lists which folders agents may touch); if you don't see the Agent toggle do anything, it isn't configured on your instance.

Turn on the Agent icon (the robot button) and ask for something that needs a real workspace — "in ~/Work/myproj, add a failing test for the parser and make it pass — delegate it." The assistant writes a self-contained brief and a confirmation box appears showing the folder and the brief. Nothing runs until you approve, and you can edit the brief first — the agent sees exactly what you approve. The agent uses the same model your chat is using.

While it works you'll see a live activity card with a Stop task button; when it finishes, a collapsed card records the brief, what it did, a short summary, and a list of the files it changed — saved with the message so it survives a reload. If you deny it, stop it, or it times out, the assistant simply answers with what it has. The agent can only ever work inside the folders your instance allows, one task at a time.

On a shared (multi-user) instance the operator usually configures managed workspaces: each conversation gets its own private folder, created the first time you delegate and reused for follow-ups in the same chat — so "now add sound effects" continues where the last task left off. Other users can never see your folders. In this mode you don't pick a directory; the confirmation box shows the one assigned to the chat.

Getting at your files (SFTP)

If your instance has SFTP enabled (managed workspaces only), you can browse and copy everything the agent built — and drop files in for it to work on next — using any SFTP client. Sign in with your Omni Chat username and password; you only ever see your own folders, one per conversation. Each run card shows that conversation's folder name (cht_…) next to two copy buttons: one copies the folder's path on the server, the other (the server icon, shown only when SFTP is on) copies a ready-to-paste sftp://you@your-server:2222/cht_… address — drop it into any of the clients below and you land straight in that conversation's folder. Paste it into an SFTP client or the Terminal, not a web browser — browsers can't open sftp:// links, so Safari or Chrome will show an "invalid address" error.

• VS Code: install the SSH FS extension and add a connection to sftp://you@your-server:2222 — your workspaces mount like a local folder. Note it must be SSH FS, not Microsoft's Remote - SSH: Remote - SSH needs to run a server program on the remote machine, and this port deliberately allows file access only — no commands.
• FileZilla / Cyberduck: protocol SFTP, host your-server, port 2222, password login.
• Terminal: sftp -P 2222 you@your-server.

The port may differ on your instance — ask whoever runs it. There is no shell login on this port, just file access.

Generating images

If your instance has an image backend configured, just ask: "Generate an image of a cyberpunk cat riding a skateboard." A tool-capable model writes a detailed prompt (you can ask for square, portrait, or landscape) and an animated placeholder appears in the response the moment generation starts, already shaped like the image to come. With the bundled backend you then watch the image form: a blurred preview of the actual picture appears part-way through and sharpens as the percentage climbs, until the finished image replaces it in place, with the prompt as its caption. Generation happens on the server's configured backend and can take from seconds to a couple of minutes depending on the hardware — the response simply continues once it lands.

Each image has three buttons: Download saves it (named after the prompt), Variation drops a ready-made variation request into the composer — edit it if you like, then send, and the model generates a fresh take — and Edit starts an image-to-image edit of that exact picture (see below). Images are stored with the conversation (they survive reload and appear in chat exports) and are private to your account. The assistant can produce at most two images per response; if the backend is offline the assistant says so and answers without one. The image server draws one picture at a time — if someone else's generation is running, the placeholder reads "Waiting for the image server… (N ahead)" until it's your turn (waiting never counts against the timeout), and when the wait queue is full the assistant simply reports that the server is busy and suggests trying again shortly. If nothing happens when you ask for images, the instance has no image backend configured — see Running Omni Chat, combination g.

Editing images (image-to-image). The assistant can also transform an existing picture: attach a photo and say "make this look like a watercolor", say "now make it night time" about an image it just generated, or click Edit on any generated image — it prefills the message with that image's reference so you just type the change. While an edit runs, the placeholder starts from a blurred copy of the source image, and the result carries an Edited caption. This works with any tool-capable chat model — the chat model does not need vision; it hands your instruction and an image reference to the image backend, which does the actual transformation. (If your instance's image backend runs without a dedicated edit model, edits are re-imaginings guided by your instruction rather than surgical changes — style changes land better than element edits, and describing the full desired result works better than a terse instruction. Ask whoever runs it about IMAGEGEN_EDIT_STRENGTH for a stronger restyle, or IMAGEGEN_EDIT_MODEL_ID for precise instruction edits.)

Chapter 6

Models & Backends

Choosing a model

Pick a model from the Model dropdown in the composer row or in the Settings panel. The list is fetched from your backend's /v1/models endpoint. Each chat remembers both its model and its backend, so reopening a chat restores the right one.

You can't switch models mid-stream — the picker is disabled while a response is in flight. Use Stop (or regenerate afterwards) to change models.

A small status LED sits beside each real model inside the composer Model picker, and the closed picker repeats the selected model's LED next to its name. Green means the selected backend confirms the model is loaded and ready; amber means it is loading; neutral means it is unloaded, sleeping, or the backend does not expose a trustworthy signal. The Auto (Smart Router) entry and the backend divider rows carry no LED. Omni Chat refreshes the state while the app is visible. It auto-detects Ollama, LM Studio, llama.cpp (single-server and router modes), vLLM, and oMLX status APIs. A failed optional status check never removes an otherwise available model.

Combining multiple backends

Omni Chat can present models from several OpenAI-compatible backends in one picker — for example a local vLLM/LiteLLM proxy alongside a hosted provider. The dropdown groups models under a divider per backend; choosing a divider selects that backend's first model. A backend that fails to load at startup is marked offline and skipped — the others keep working.

This is configured with an endpoints.json file, covered in Chapter 11. With no such file, the app uses CHAT_ENDPOINT_URL, which may itself be a single URL or a comma-separated list.

Auto (Smart Router)

Whenever at least one real model is available, the Model dropdown also offers a distinct ✨ Auto (Smart Router) entry pinned above the per-backend groups. Select it and every message you send is automatically classified and routed to the best available model for that turn — weighing the task type (coding, reasoning, a quick question), the capabilities the turn needs (tool calling for web search, a thinking channel), and the request's complexity and size, so heavier or larger-context turns favor bigger models and trivial ones favor faster models. No manual model-switching needed. Auto is opt-in: it is never the default for a new chat.

While a chat is on Auto, the composer shows Auto → <model> once a response has routed, and the assistant message's Generation details panel shows which model and backend handled it, the routing reason, and how long the routing decision took. Because the concrete model — and therefore its context window — is chosen per message, the context meter is hidden while Auto is selected.

Routing also weighs each model's resolved capabilities — vision, tool calling, parameter size, and vendor/family — taken from a built-in catalog of common model ids, from id inference, or from a per-endpoint model_overrides array in endpoints.json. A message with an image attachment is routed to a vision-capable model whenever one is available.

Auto works out of the box with a built-in policy. Power users can customize model tagging and per-task preferences with a router.json file — see Chapter 11 for the full schema and how tagging and routing decisions work.

Chapter 7

Profiles

A profile is a one-click starting point. Selecting one applies a persona prompt plus a bundle of style settings (tone, length, creativity, and format) for the current session.

Built-in profiles

Selecting a profile

The welcome screen shows up to eight profile cards total; every configured profile is also in the Profile dropdowns. The synthetic ✨ Auto entry described below is shown first when present and counts as one of the eight cards. Pick a card to apply its persona and style.

You can define your own profiles to extend or override the built-ins — see Chapter 11.

Auto profile

The Profile dropdown also offers a synthetic ✨ Auto entry. auto is a reserved id — you cannot define your own profile with that id. Picking Auto is an autopilot: it also switches the chat to the Auto (Smart Router) model (when that model is available), so one choice hands both the backend model and the per-turn persona to the router. Auto only picks a persona when the chat's model is Auto (Smart Router). On such a turn the profile catalog is offered to the router: the semantic router classifier, when configured in schema mode and answering in time, picks the best-fitting profile; otherwise (Arch-Router, or the classifier disabled or too slow) a deterministic task-affinity heuristic matches the turn's task to a profile whose description reads like that kind of work. Either way that profile's persona and style are used and the response is labeled Auto → <Profile Label>, just like a manually-picked profile. When no profile fits (e.g. a trivial greeting) or the chat uses a manually-selected model, Auto behaves exactly like No profile and the response is simply labeled Auto.

Chapter 8

Response Controls

The Settings panel shapes each response. These are session-level controls; a profile simply sets several of them at once.

Tone

Single-select: Default, Professional, Friendly, Direct, Executive. Controls tone only. Default adds no tone instruction.

Response Length

Single-select target for the output budget:

Creativity

Single-select: Precise, Balanced, Exploratory. This always changes the style prompt. It sends temperature and top_p sampling parameters only when the backend is configured to honor sampling (CHAT_HONORS_SAMPLING=true, or honors_sampling on the endpoint). Otherwise those parameters are intentionally omitted.

Format

Single-select shape for the output: Natural, Structured, Bullets, Step-by-step, Table-first.

Response budget Advanced

A per-session override for the output cap — the maximum tokens the assistant may use for the next response. Leave it empty to use the response-length preset and automatic clamping. Any value is still clamped to the backend's max_output_tokens if one is set.

Session instructions

Omni Chat assembles a single upstream system message from, in order: app safety invariants, hidden real-time context, assistant soul, editable session instructions, retrieved memory, retrieved sources, and summaries. The panel shows only the editable session instructions generated from project/profile/style settings; managed safety and SOUL are hidden.

Chapter 9

Memory

Memory lets Omni Chat remember durable facts, preferences, and decisions across turns and chats — separate from the raw message history.

The Memory toggle

• On (default) — the assistant can save durable memories and recalls the relevant ones on later turns, injected into its prompt.
• Off — chat history is still stored, but nothing is written to or read from durable memory for that chat.

The toggle lives in the top-right user menu and is remembered per chat.

Asking the assistant to remember

With Memory on, just tell the assistant — for example, "Remember that I'm vegetarian" — and it saves a durable memory; a small chip confirms "Memory saved". In a later chat, ask "What kind of food do I eat?" and it answers from that memory without being told again. Say "Forget that I'm vegetarian" to remove it.

Memory kinds

preference, fact, decision, constraint.

Scopes

Managing memories

Open Manage Memories from the top-right user menu to see everything the assistant remembers about you. From there you can filter by scope, edit a memory's text or kind, pin the ones that should always be kept in context, add a memory manually, delete any of them, and see each one's provenance — the chat it came from. Your memories are private to your account.

Smarter recall (optional)

If the operator has configured an embeddings endpoint (CHAT_EMBEDDINGS_URL and friends), memory recall becomes semantic: the assistant surfaces the memories most relevant to what you're currently asking, and saving something you already mentioned — even in different words — updates the existing memory instead of creating a near-duplicate. Without an embeddings endpoint, recall falls back to most-recent-first and duplicate detection is exact-text; either way, memory works.

Automatic memories

If the operator has configured the auxiliary model (CHAT_AUX_URL and CHAT_AUX_MODEL), the assistant quietly captures durable facts you mention — lasting preferences, your ongoing projects, decisions, and constraints — without you having to say "remember this". This runs in the background after a reply, so it never slows a conversation, and it uses a separate small model rather than your main chat model. It is conservative by design: it saves only clearly-stated, lasting facts, skips sensitive data (passwords, card or account numbers, secrets), and won't create duplicates of things you already told it. Every automatic memory shows up in the Memories panel tagged by where it came from, so you can review, edit, or delete it. Auto-capture is off unless the operator turns it on, and you can disable it per chat with the memory save policy.

Staying coherent in long chats (automatic context compaction)

When a conversation gets very long, it eventually exceeds the model's context window and the oldest messages would normally fall out of view. Instead, the assistant keeps a short rolling recap of the earlier part of the chat, refreshed quietly in the background as the conversation grows — each refresh folds the new turns into the previous recap, so even the very beginning of a long chat is never forgotten. When old messages no longer fit, the assistant sees the "Earlier in this conversation…" recap in their place, so it stays on topic instead of losing the thread. This works even without the optional auxiliary model: the recap is then generated by the chat's own model between turns, and it never slows down a reply.

Your transcript is never changed — you can always scroll back through the full conversation. When compaction was active on the latest response, a subtle dashed divider marks the boundary: messages above it reached the model only as the recap. Click View summary on the divider to read exactly what the model is told, and the response's info popup shows a "Context: Compacted" row. You can still edit, regenerate, or delete any message, including ones above the boundary — doing so simply discards the now-stale recap and rebuilds it in the background a turn later.

Tidying up over time

As your memories accumulate, the assistant occasionally tidies them in the background (with the same optional memory model, rate-limited so it runs at most once in a while). It merges duplicates, retires memories that a newer one has contradicted or made obsolete, and distills recurring themes into higher-level "insight" memories. Your pinned memories are never touched. Nothing is ever permanently erased: a retired memory keeps a link to the one that replaced it, so the whole history stays auditable and reversible. The result is that a long-lived memory set gets sharper and less cluttered over time instead of filling up with near-duplicates.

Remembering across conversations

Sometimes the thing you need was discussed in a different chat entirely. When cross-chat recall is available (it needs the embeddings model configured), the assistant can pull in a short recap of your other conversations that are relevant to what you're asking now — shown to it as "From earlier conversations…". So if you start a fresh chat and ask about a project, decision, or topic you worked through days ago, it can pick up the thread instead of starting from nothing. It only ever draws on your own chats, surfaces just the few most relevant ones, and follows the per-chat Memory switch — turn Memory off for a chat and no past-conversation context is pulled in.

Chapter 10

Sources & Retrieval

Retrieval grounds answers in documents you've added, attaching source cards that show which documents and chunks were used.

Include Sources

Card visibility applies to both indexed-document and web-search results. Turning the switch off does not delete saved references; turning it back on reveals historical cards again.

Adding documents

Attach documents via upload or drag-and-drop. Once indexed, they become available to retrieval. Retrieval requires an embeddings endpoint to be configured (see Chapter 11).

Chapter 11

Configuration Reference

Omni Chat is configured through environment variables set at startup, plus two optional JSON files for backends and profiles.

Environment variables — network & storage

Environment variables — backend & model

Reasoning models often have endpoint-specific parameters. Omni Chat infers common adapters such as Qwen served by llama.cpp and DeepSeek V4 served by ds4.c. For OMLX endpoints, Omni Chat reads the optional /v1/models/status metadata and enables the switch when thinking_default is either true or false; both values mean the model template supports enable_thinking. Custom endpoints can still define thinking request bodies in endpoints.json; the switch sends on_body for auto and off_body for off. Models with no inferred or configured adapter keep the switch disabled.

Environment variables — embeddings

Environment variables — auxiliary model

One optional small, fast, non-thinking model powers the background intelligence: chat-title generation, memory extraction, rolling summaries, consolidation, and — when no dedicated classifier is set — smart-router classification. It runs off the response path and never uses the main chat model. (Rolling chat summaries are the one exception when no auxiliary model is configured: they then fall back to the chat's own model between turns, so long-chat compaction always works.) A 1–2B non-reasoning instruct model is ideal — the recommended default is LFM2.5-1.2B-Instruct (Liquid AI, ~731 MB at Q4_K_M), served with llama-server -hf LiquidAI/LFM2.5-1.2B-Instruct-GGUF:Q4_K_M; see .env.example for smaller and larger alternatives.

Environment variables — config locations

Environment variables — semantic router classifier

Environment variables — web search

Search uses structured function calls and persists real results as cards. Snippets are untrusted summaries, not full-page content. Include Sources does not enable or disable web search; it controls indexed-document retrieval and whether saved document, web-search, and web-fetch cards are visible. Each assistant response records whether web search was unavailable, available but unused, successful, successful with no results, or failed; the Generation details panel reports that outcome even while cards are hidden.

With CHAT_WEB_FETCH=true a companion fetch_url tool lets the model retrieve one specific URL — a web page, article, or (via a capable reader) a YouTube transcript — extract its readable text, and summarize or quote it. The direct provider fetches in-core behind an SSRF guard that refuses private, loopback, link-local, and cloud-metadata addresses; the reader provider delegates to an external or self-hosted reader endpoint for JS-heavy or anti-bot sites. Fetched content is treated as untrusted, persists as a source card (subject to the same Include Sources visibility as web-search cards), and its outcome is folded into the single Web access line of the Generation details panel alongside web search (that line reads Used whenever either tool ran). web_search and fetch_url share one bounded tool-call budget per turn. When the fetch tool is enabled and your message contains a URL, the core fetches it automatically before the model runs, so summarizing a pasted link works even if the model would otherwise try to search for it.

Every turn also carries a hidden, system-injected context block (separate from editable Session Instructions): the current date and time in your browser's timezone, plus a short directive. When the web_search tool is available the directive tells the model to search for current or possibly-changed facts and to trust fresh results over its own memory; the fetch_url tool adds guidance to read a specific URL for summarizing. When neither web tool is available it instead tells the model it has no web access and should flag that its knowledge may be out of date. The browser sends only its IANA timezone name; the server keeps the clock and falls back to UTC when the name is missing or invalid. No configuration is required.

With CHAT_IMAGE_GEN_URL set, tool-capable models are offered a generate_image tool (no per-chat toggle). The core calls the configured backend over the OpenAI Images API, stores the produced image with the conversation (bytes never enter the model context), and streams it to the chat as it lands — including live progress and blurred partial previews when the backend supports OpenAI's partial_images streaming (the bundled sidecar does; those progress events also reset the CHAT_IMAGE_GEN_TIMEOUT idle deadline, so long generations don't time out while they're visibly working). The tool has its own budget of two images per response, separate from the web tools; the assistant response's Generation details panel records an Image generation outcome (used, failed, or not used). The bundled backend is the Z-Image-Turbo sidecar (compose.imagegen.yaml, GPU required) — see Running Omni Chat, combination g — but any server implementing POST /v1/images/generations with b64_json works.

Multiple backends — endpoints.json

Create <config-dir>/endpoints.json to combine backends. Each entry needs a name and base_url. An API key is optional and may be inline (api_key) or read from an env var (api_key_env). Per-endpoint capability fields mirror the CHAT_* model settings and apply only to that backend, including optional thinking adapter overrides and model-pattern overrides.

{
  "schema_version": 1,
  "default_endpoint": "local-litellm",
  "endpoints": [
    {
      "name": "local-litellm",
      "base_url": "http://localhost:4000",
      "thinking": {
        "on_body": { "reasoning_effort": "medium" },
        "off_body": { "reasoning_effort": "none" },
        "parse_think_tags": true
      }
    },
    {
      "name": "openai",
      "base_url": "https://api.openai.com",
      "api_key_env": "OPENAI_API_KEY",
      "max_tokens_field": "max_completion_tokens",
      "context_window": 128000,
      "max_output_tokens": 16000,
      "honors_sampling": true
    },
    {
      "name": "vllm-local",
      "base_url": "http://localhost:8000",
      "api_key": "sk-local-only-not-a-real-secret",
      "context_window": 32768,
      "model_overrides": [
        { "model_pattern": "qwen2.5-vl*", "vision": true, "tool_calling": true, "params_b": 7 }
      ]
    }
  ]
}

Assistant soul — SOUL.md

The built-in assistant soul makes Omni Chat warm, thoughtful, conversational, and lightly playful by default. To customize it, create <config-dir>/SOUL.md or point CHAT_SOUL_PATH at a Markdown file. A missing default file is fine and uses the built-in soul; an explicit path must exist and must not be empty. SOUL is personality text, not a safety boundary: the app safety invariants are still hard-coded and injected before the single upstream system message is sent.

Custom profiles — profiles.json

Extend or override the built-in profiles by creating <config-dir>/profiles.json.

{
  "schema_version": 1,
  "profiles": [
    {
      "id": "review_code",
      "label": "Review Code",
      "description": "Find correctness, maintainability, security, and test gaps.",
      "persona_prompt": "Review code for correctness, maintainability, edge cases, security and privacy risks, and missing tests. Prioritize concrete findings...",
      "icon": "code",
      "style": {
        "tone": "direct",
        "response_length": "detailed",
        "creativity": "precise",
        "format": "structured"
      }
    }
  ]
}

Custom router policy — router.json

Customize how the Auto (Smart Router) model tags and picks models by creating <config-dir>/router.json.

{
  "schema_version": 1,
  "routes": [
    { "name": "code", "description": "writing, debugging, or reviewing code or technical artifacts" }
  ],
  "tag_rules": [
    { "pattern": "*coder*", "tags": ["coder", "code"] },
    { "pattern": "*mini*", "endpoint": "openai", "tags": ["small", "fast"] }
  ],
  "policies": [
    {
      "task": "code",
      "prefer": [
        { "tags": ["coder"], "min_context": 32768 },
        { "tags": ["general"] }
      ]
    }
  ]
}

How Auto tags and routes models

Every available model gets a set of lower-snake_case tags before Auto picks one for a turn. Tags come from three layers, and later layers only add tags — none of them remove a tag a candidate already has:

1. Built-in glob rules match substrings of the model id: *coder* / *codestral* / *devstral* / … → coder; *deepseek-r1* / *qwq* / *reason* / *o1* / *o3* / *gpt-oss* → reasoning; *mini* / *small* / *phi* / *1b*…*8b* → fast; *70b* / *large* / *opus* / *gpt-4* / *gpt-5* → large.
2. Computed tags are attached automatically, not from glob rules: general (every candidate), large_context (context window ≥ 64K), and capability tags vision, reasoning, size_small / size_medium / size_large (the small and large ends also alias to fast / large), and vendor_<v> / family_<f>. These resolve strongest-first from a per-model model_overrides entry in endpoints.json, a built-in catalog of common model ids, then inference from the id itself.
3. Your own tag_rules[] (above) add more tags the same way the built-ins do — a model's final tag set is the union of every matching rule.

For each turn, Auto classifies a task (code, reasoning, simple_fast, creative, or general), then narrows the candidates: models that can't fit the estimated input are dropped (unless that would empty the list), and an image attachment keeps only vision-tagged models when one exists. It then applies soft boosts — favoring fast models for low-complexity turns, large/reasoning for high-complexity ones, large_context when the turn needs it, and tool-calling/thinking-capable models per your toggles — before walking the task's policies[].prefer list top to bottom and picking the first selector that still matches at least one candidate. If nothing in prefer ever matches, the default endpoint's first remaining candidate wins.

A user policy for a task replaces the built-in policy for that task outright rather than merging with it, so writing your own prefer list is the reliable way to pin a task to a specific model — for example, to keep creative turns on one backend model even though a built-in glob like *large* or *opus* would otherwise tag it (and win) differently:

{
  "schema_version": 1,
  "tag_rules": [
    { "pattern": "*my-writer-model*", "tags": ["storyteller"] }
  ],
  "policies": [
    {
      "task": "creative",
      "prefer": [
        { "tags": ["storyteller"] },
        { "tags": ["general"] }
      ]
    }
  ]
}

This tags one specific backend model storyteller and makes it the first choice for creative turns, falling back to any available model (general) if that one is offline. See docs/router.example.json for a fuller starter covering the classifier, routes, and multiple policies together, and docs/endpoints.example.json for the model_overrides syntax used to set capability tags explicitly per backend.

Semantic router classifier

By default Auto classifies each turn with fast, deterministic heuristics only. To classify turns semantically instead, either configure the auxiliary model (CHAT_AUX_*) — which automatically doubles as the classifier via the JSON-schema adapter with no extra setup (disable with CHAT_AUX_ROUTER=0) — or point the router at a dedicated classifier: set CHAT_ROUTER_CLASSIFIER_URL and CHAT_ROUTER_CLASSIFIER_MODEL (and optionally CHAT_ROUTER_CLASSIFIER_API_KEY / CHAT_ROUTER_CLASSIFIER_TIMEOUT), or add an equivalent classifier section to router.json (see the commented example in docs/router.example.json) — env vars override the file per field. A dedicated classifier always takes precedence over the auxiliary model.

The default classifier model is Arch-Router-1.5B, a purpose-built router model. The core auto-detects it and speaks its native protocol: it is handed the task classes (with descriptions overridable via routes above) and returns the single best-fitting one, which maps to a model through the usual tag rules and policies. classifier.format / CHAT_ROUTER_CLASSIFIER_FORMAT forces the adapter — set schema to use a general instruct model with the richer JSON-schema classification instead. Because Arch-Router returns only a route, under it (or with the classifier disabled) the Auto-profile persona is chosen by a deterministic task-affinity heuristic over the profile catalog rather than by the classifier; the schema classifier gives a richer, content-aware persona pick.

On every routed turn the classifier races the heuristics against a deadline (default 3000ms, sized for a CPU-only sidecar classifying a fresh, full-context turn). Whichever finishes first within the deadline wins; a classifier error, timeout, or unset config falls back to heuristics, so a turn never fails or blocks on it. The persisted routing reason records which one won, prefixed semantic: or heuristic: , and the per-message stats hover's decision time includes any time spent waiting on the classifier.

To self-host a tiny classifier model with no extra setup, use the compose.router.yaml (or compose.router-only.yaml for host-run cores) Compose overlay described under Docker Compose deployment; it runs ghcr.io/ggml-org/llama.cpp with the Arch-Router-1.5B GGUF and constrained JSON output.

Chapter 12

Administration & Deployment

Multi-user isolation

Every owned resource carries an owner, and the store layer filters all queries by the session's user. Identity comes from the session, never the client. Accessing another user's private resource returns 404 — the app reveals nothing about resources you don't own.

Projects have a visibility of private or shared. Marking a project shared makes its chats, documents, and memory readable and writable by members of the group; private projects stay yours alone.

Deployment modes

Production build & run

Build the web SPA and the core binary (the SPA is embedded into the binary):

make build
# binary: .cache/bin/omni-chat-core

Run it with a persistent database path and your endpoint:

CHAT_ENDPOINT_URL=http://your-endpoint:port \
  CHAT_BIND=0.0.0.0 \
  CHAT_PORT=8080 \
  CHAT_DB_PATH=/persistent/path/app.db \
  ./.cache/bin/omni-chat-core

Docker Compose deployment

To include the optional self-hosted SearXNG provider, use:

printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" >> .env
docker compose -f compose.yaml -f compose.search.yaml up -d

The overlay enables JSON output, connects Omni Chat over the private Compose network, and exposes SearXNG only at 127.0.0.1:8888 by default. The generated SEARXNG_SECRET is required, belongs in the ignored .env file, and must not be committed or copied into settings.yml.

For a host-run core (make dev) instead of the full Compose stack, start just SearXNG and point the core at its published port:

printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" >> .env
docker compose -f compose.search-only.yaml up -d
CHAT_WEB_SEARCH_URL=http://localhost:8888 make dev

To include the optional llama.cpp auxiliary-model sidecar (see Auxiliary model above), use:

docker compose -f compose.yaml -f compose.aux.yaml up -d

The overlay wires CHAT_AUX_URL / CHAT_AUX_MODEL to an internal aux-model service running ghcr.io/ggml-org/llama.cpp, which downloads the configured Hugging Face GGUF (default LiquidAI/LFM2.5-1.2B-Instruct-GGUF, ~731 MiB) into a named volume on first run. This one model handles chat titles, memory jobs, and smart-router classification, so on its own it needs no separate router sidecar. It is stackable with the other overlays, e.g. docker compose -f compose.yaml -f compose.search.yaml -f compose.aux.yaml up -d. For a host-run core, start just the sidecar and point the core at its published port:

docker compose -f compose.aux-only.yaml up -d
CHAT_AUX_URL=http://localhost:8091 \
CHAT_AUX_MODEL=lfm2.5-1.2b-instruct make dev

To also run the optional dedicated llama.cpp router-classifier sidecar (see Semantic router classifier below), use:

docker compose -f compose.yaml -f compose.aux.yaml -f compose.router.yaml up -d

The overlay wires CHAT_ROUTER_CLASSIFIER_URL / CHAT_ROUTER_CLASSIFIER_MODEL to an internal router-classifier service running ghcr.io/ggml-org/llama.cpp, which downloads the configured Hugging Face GGUF (default katanemo/Arch-Router-1.5B.gguf, ~1 GiB) into a named volume on first run — allow a few minutes before the sidecar reports healthy and Omni Chat starts. When present, this dedicated classifier outranks the auxiliary model for routing (the aux model then does titles and memory only); omit it and the aux sidecar classifies via the fallback. For a host-run core, start just the sidecar and point the core at its published port:

docker compose -f compose.router-only.yaml up -d
CHAT_ROUTER_CLASSIFIER_URL=http://localhost:8090 \
CHAT_ROUTER_CLASSIFIER_MODEL=arch-router-1.5b make dev

To run the fetch_url tool through a self-hosted reader sidecar instead of the in-core direct provider, use:

docker compose -f compose.yaml -f compose.fetch.yaml up -d

The overlay enables the tool (CHAT_WEB_FETCH=true), switches it to the reader provider, and points Omni Chat at an internal reader service on http://reader:8081 (also published on 127.0.0.1:3001 for debugging). The service is ghcr.io/jina-ai/reader:oss, Jina AI's official self-host image (multi-platform: linux/amd64 and linux/arm64, so it runs unmodified on Apple Silicon) that renders JavaScript in a bundled headless browser; override READER_IMAGE/READER_VERSION to pin or replace it. It is stackable with the other overlays. When to use it: the default direct provider needs no extra container and handles most static or server-rendered pages, but it only does a plain HTTP GET — it cannot run JavaScript, is blocked by anti-bot defenses such as Cloudflare, and cannot read YouTube transcripts. The reader sidecar runs a real browser, so it handles all three, at the cost of a much larger image, more memory, slower fetches (consider raising CHAT_WEB_FETCH_TIMEOUT, up to 60s), and sending target URLs to that container. Prefer direct and switch to the reader overlay only when you need one of those capabilities. For a host-run core, start just the reader and point the core at its published port:

docker compose -f compose.fetch-only.yaml up -d
CHAT_WEB_FETCH=true CHAT_WEB_FETCH_PROVIDER=reader \
CHAT_WEB_FETCH_URL=http://localhost:3001 make dev

To offer the generate_image tool, use the image-generation overlay (see Running Omni Chat, combination g, for the requirements — an NVIDIA GPU with the container toolkit, and a ~12 GB first-run weight download into the imagegen-models volume):

docker compose -f compose.yaml -f compose.imagegen.yaml up --build -d

The overlay builds the bundled Z-Image-Turbo sidecar (docker/imagegen/ — FastAPI + diffusers exposing POST /v1/images/generations) and sets CHAT_IMAGE_GEN_URL=http://imagegen:8000 on the app. Tune it with IMAGEGEN_MODEL_ID (any diffusers text-to-image model id; default Tongyi-MAI/Z-Image-Turbo), IMAGEGEN_STEPS, and — for precise instruction edits via the edit_image tool — IMAGEGEN_EDIT_MODEL_ID (recommended: Qwen/Qwen-Image-Edit-2509, Apache-2.0 but ~20B; alternative: black-forest-labs/FLUX.1-Kontext-dev, 12B, gated + non-commercial — accept its license on Hugging Face and set HF_TOKEN in .env; unset falls back to img2img with the base model, tuned by IMAGEGEN_EDIT_STRENGTH, default 0.7 — raise toward 0.8 for stronger restyles — and IMAGEGEN_EDIT_GUIDANCE, default 1.0, which only helps CFG-capable models as the distilled base can burn above 1) in .env. Tongyi's announced Z-Image-Edit has no public weights as of July 2026. On non-x86 GPU hosts (Jetson/L4T or arm64 CUDA), override IMAGEGEN_BASE_IMAGE with a matching CUDA-enabled PyTorch base and set IMAGEGEN_RUNTIME=nvidia — see the GPU note in Getting started. For a host-run core — or to run the sidecar on a separate GPU machine — start just the sidecar and point the core at its published port (127.0.0.1:8001 by default; set IMAGEGEN_PUBLISH_ADDRESS=0.0.0.0 and an IMAGEGEN_API_KEY/CHAT_IMAGE_GEN_API_KEY pair when it must be reachable over the network):

docker compose -f compose.imagegen-only.yaml up --build -d
CHAT_IMAGE_GEN_URL=http://localhost:8001 make dev

Any other OpenAI-Images-compatible server also works — set CHAT_IMAGE_GEN_URL (plus CHAT_IMAGE_GEN_MODEL/CHAT_IMAGE_GEN_API_KEY as needed) and skip the sidecar.

To use the default in-core direct provider under the base Compose stack (no sidecar), just add CHAT_WEB_FETCH=true to .env — no overlay needed. If you are testing a feature (such as fetch_url) that predates the published latest image, docker compose up -d alone keeps running that stale image; rebuild from your checked-out source with docker compose up --build -d to pick up local changes.

The repository's Compose configuration pulls the non-root, multi-platform image from the GitLab Container Registry, publishes the app only on 127.0.0.1:8080 by default, checks /api/health, and restarts the service unless it is stopped explicitly:

# Optional environment overrides; .env is ignored by Git.
cp .env.example .env
docker compose up -d
docker compose ps

Compose passes variables from .env into the container. Use it for CHAT_ENDPOINT_URL, CHAT_API_KEY, variables referenced by api_key_env, and other supported CHAT_* overrides. Set OMNI_CHAT_PUBLISH_ADDRESS=0.0.0.0 only when the app should be reachable from other machines, and protect the published port appropriately.

The default image is registry.gitlab.com/geekaholic/omni-chat:latest for both AMD64 and ARM64. Every successful main-branch pipeline also publishes an immutable full-commit-SHA tag. Set OMNI_CHAT_IMAGE=registry.gitlab.com/geekaholic/omni-chat:<full-commit-sha> in .env to pin a deployment. Private projects require docker login registry.gitlab.com before pulling. Use docker compose up --build -d to build from the local checkout instead.

Connecting a container to AI on the host

localhost inside a container is the container itself. Use host.docker.internal for an OpenAI-compatible endpoint running on the Docker host. Compose adds the host-gateway mapping needed on native Linux; Docker Desktop provides the same hostname. On native Linux, the AI process must also listen on an interface reachable from Docker's bridge rather than only 127.0.0.1. Bind the AI server to an appropriate host interface and restrict that port with host firewall rules.

Docker data and configuration

SQLite is stored in the named omni-chat-data volume. Operator-managed JSON files are bind-mounted read-only from ./config to /config, so they remain editable and survive image upgrades:

cp docs/endpoints.docker.example.json config/endpoints.json
cp docs/profiles.example.json config/profiles.json
cp docs/SOUL.example.md config/SOUL.md
# Edit the files, then reload startup configuration.
docker compose restart

The Docker endpoint example points to host.docker.internal:8000. If config/endpoints.json exists, it takes precedence over CHAT_ENDPOINT_URL. Existing JSON configuration is validated at startup exactly as it is for native runs.

Update and replace the container without deleting its data:

git pull
docker compose pull
docker compose up -d

For a consistent backup, stop writes while archiving the volume:

docker compose stop
mkdir -p backups
docker run --rm \
  -v omni-chat-data:/data:ro \
  -v "$PWD/backups:/backup" \
  alpine:3.22 tar -czf /backup/omni-chat-data.tar.gz -C /data .
docker compose start

Docker without Compose

docker volume create omni-chat-data
docker run -d --name omni-chat --restart unless-stopped \
  --add-host host.docker.internal:host-gateway \
  -p 127.0.0.1:8080:8080 \
  -e CHAT_ENDPOINT_URL=http://host.docker.internal:8000 \
  -v omni-chat-data:/data \
  -v "$PWD/config:/config:ro" \
  registry.gitlab.com/geekaholic/omni-chat:latest

For a local source build, run docker build -t omni-chat:local . and use omni-chat:local as the final argument instead.

Importing from ChatGPT

Bring existing ChatGPT history into Omni Chat with the core import chatgpt CLI subcommand. In ChatGPT, go to Settings → Data controls → Export data; a download link for the export zip arrives by email.

./.cache/bin/omni-chat-core import chatgpt ~/Downloads/chatgpt-export.zip --user alice

<export> accepts the export zip as downloaded, an already-unzipped export directory, or a bare conversations.json. A bare JSON file has no attachments to pull from, so the importer prints a note to stderr and continues without them. Both the classic single-conversations.json layout and the newer sharded layout (conversations-000.json, conversations-001.json, … plus .dat assets with a conversation_asset_file_names.json restoring their original names) are supported.

For each conversation the importer takes the active branch only — exactly what was last seen in ChatGPT, not every edited-away alternative. It imports the title, timestamps, user/assistant messages, reasoning ("thoughts") folded into the following assistant message, images/files as attachments, and archived state; every imported chat gets a chatgpt tag. ChatGPT memories (bio-tool writes) are harvested into the user's user-scope memories, deduplicated against what's already there. Tool-call plumbing (code execution, browsing traces, canvas) is not imported and is counted in the summary.

Re-running the import against a newer export is incremental: conversations that haven't changed are skipped, conversations that grew in ChatGPT get only their new messages appended, and conversations whose active branch changed (an edit that switched branches) are skipped with a warning — existing messages are never rewritten or deleted. It is safe to import while the server is running (SQLite WAL). Imported chats have no model/endpoint set, so they pick the user's default model the next time they use them.

Importing under Docker Compose

When Omni Chat runs via docker compose, the same binary is the container's entrypoint, so the import runs as a one-off container that shares the service's database volume and environment:

docker compose run --rm \
  -v "$PWD/chatgpt-export.zip:/import/export.zip:ro" \
  omni-chat import chatgpt /import/export.zip --user alice

docker compose run reuses the service's CHAT_DB_PATH=/data/app.db and the omni-chat-data volume, so no --db flag is needed and the import lands in the same database the server uses — safe to run while the service is up. Bind-mount the export read-only at a fresh path such as /import/… (avoid /tmp, which the service config covers with a small tmpfs). All the flags above work the same way; add --dry-run first for a report before writing anything. If the image predates the import feature, rebuild it with docker compose build.

Quality gate

For contributors: make check runs fmt + lint + test + build and must pass before a change is considered done.

Chapter 13

Troubleshooting & FAQ

No models appear in the picker

The model list comes from the backend's /v1/models. Confirm CHAT_ENDPOINT_URL is set and reachable, or that your endpoints.json is valid. Remember that endpoints.json, if present, makes CHAT_ENDPOINT_URL ignored. A backend that fails at startup is shown as offline and skipped.

No models appear when Omni Chat runs in Docker

Do not use localhost for an AI server running on the host. Set the endpoint to http://host.docker.internal:<port>. If you mounted config/endpoints.json, update its base_url because that file overrides CHAT_ENDPOINT_URL, then run docker compose restart. On native Linux, also confirm the AI server listens on a bridge-reachable interface.

SearXNG exits because server.secret_key is unchanged

SearXNG rejects its bundled ultrasecretkey. Run printf 'SEARXNG_SECRET=%s\n' "$(openssl rand -hex 32)" >> .env, then run docker compose -f compose.yaml -f compose.search.yaml up -d again. Compose reports a clear setup error before startup when the variable is missing.

Empty model picker with a LAN endpoint on macOS

If CHAT_ENDPOINT_URL points to a machine on your local network (e.g. http://192.168.x.x:8000 or http://<hostname>.local:port) and the picker stays empty even though curling the same /v1/models URL works, the cause is usually macOS Local Network privacy. Recent macOS releases block a process from reaching local-network addresses until you grant it access, which surfaces here as a "no route to host" error and an offline endpoint; public/internet endpoints are unaffected.

Fix: open System Settings → Privacy & Security → Local Network, enable the toggle for the terminal app you launch make dev from (Terminal, iTerm2, Ghostty, VS Code, …), then quit and reopen that terminal and start the core again. The entry only appears after the app has attempted a local-network connection, so run make dev once first if it isn't listed. The grant is tied to the terminal app, so it survives go run rebuilds. If you launch through tmux/screen/ssh, start from a plain terminal window so the permission is attributed correctly.

I get a context_overflow error

The input didn't fit the context window. Lower Response Length, shorten the conversation, or raise CHAT_CONTEXT_WINDOW to match your model. The current user message is never trimmed — if it alone exceeds the budget you'll see this error.

Creativity changes nothing in the model's behavior

Sampling parameters are only sent when the backend honors them. Set CHAT_HONORS_SAMPLING=true (or honors_sampling on the endpoint). Otherwise Creativity only changes the style prompt text.

The model wants a different max-tokens field

Some endpoints require max_completion_tokens. Set CHAT_MAX_TOKENS_FIELD=max_completion_tokens globally, or max_tokens_field on the specific endpoint.

No sources are ever returned

Check that Include Sources isn't Off, that an embeddings endpoint is configured, and that documents have actually been uploaded and indexed. An empty corpus never produces citations.

Reasoning parameters for thinking models

Use the UI switch first. Omni Chat detects common adapters such as Qwen on llama.cpp and capability-advertising models on OMLX. Configure thinking.on_body and thinking.off_body in endpoints.json only for custom endpoints or models with different request knobs.