Configuration

Configuration

All configuration is via environment variables (in .env) and the auth.json credential store. Most settings are optional; the router can serve model requests once at least one provider (including a local model) is configured.

Core features vs. add-ons

hermes-router splits its behavior into two groups:

• Core features — always on; they are the router. Auth, the credential pool + key rotation, failover, the circuit breaker, smart routing, protocol translation (OpenAI/Anthropic/Codex), capability probing, token estimation, request guardrails, and usage/estimated-cost tracking.
• Add-ons — optional behaviors you turn on when you want them. Each is backed by an environment variable (or some auth.json config), and unset = off (except the response cache, on by default).

hr features — see and toggle add-ons

hr features list                      # core features + every add-on, with on/off
hr features enable persistent_cache   # writes the backing var to .env
hr features disable semantic_cache
hr restart                            # apply

Add-on	Backing setting	Default	What it does
response_cache	CACHE_TTL_SECONDS	on	Serve identical requests from an in-memory TTL+LRU cache
semantic_cache	SEMANTIC_CACHE	off	Also serve cached answers for similar prompts
persistent_cache	CACHE_PERSIST	off	Mirror the cache to SQLite so it survives restarts
fast_routing	FAST_ROUTE_THRESHOLD	off	Short requests prefer low-latency providers on ties
model_discovery	AUTO_DISCOVER_MODELS	off	Refresh confirmed free chat models at startup and on demand
metrics_auth	always enabled	on	Administrator authentication on /metrics; cannot be disabled
cost_currency	COST_FX_RATE	off	Show a second currency (e.g. ₹) alongside USD spend
key_budgets	auth.json / PROXY_LIMIT_*	off	Per-key RPM / daily request / token / estimated-cost ceilings — manage with hr limit
local_model	LOCAL_BASE_URL / LOCAL_MODEL	off	Route to a model on your own machine — manage with hr model set local

hr features enable/disable toggles the simple flag add-ons by writing their variable to .env. The last two are richer config, so hr features shows their status and points you to the command that manages them. The live state is also in /v1/status under features.

Dashboard feature saves show confirmation above the add-ons. A failed or timed-out save releases the control so you can retry; use Refresh to check the saved state before retrying an uncertain request. A restart banner appears only when the server says the setting requires a restart. If an older dashboard stays on “Saving…”, reload the page to load the current dashboard files.

Where your keys live

hr auth add writes to auth.json — the router’s own credential store, kept next to the router. The repository ignores this file; do not force-add or publish it. Codex (ChatGPT subscription) logins are stored separately under codex_accounts (via hr auth import-codex); the router refreshes their OAuth access tokens automatically.

{
  "providers": {
    "openrouter": ["sk-or-key1", "sk-or-key2"],
    "gemini": ["AIzaSy-key"]
  }
}

Keys in .env (e.g. OPENROUTER_API_KEYS=k1,k2) still work too — the router reads auth.json first, then falls back to .env. Point at a different file with ROUTER_AUTH_FILE=/path/to/auth.json.

Settings (.env)

Variable	Default	Purpose
PORT	8319	Port to listen on
HOST	0.0.0.0	Bind address. Set 127.0.0.1 to listen on localhost only (recommended on a shared/VPS host — reach it via localhost or an SSH tunnel). Keep 0.0.0.0 for Docker.
PROXY_API_KEYS	(auto-generated)	Application inference keys; existing keys remain usable for inference. Dashboard key changes apply immediately and are stored in auth.json.
ADMIN_API_KEY	(auto-generated)	Separate owner credential for dashboard and global/config APIs. Never give this to an application.
HR_ENV_FILE	.env	Private environment file, also used for generated credentials.
HR_USAGE_DB	usage.db	Durable per-key usage/reservation SQLite database; preserve it across upgrades.
HR_TRUST_PROXY	0	Trust one proxy-provided scheme header only when behind a trusted reverse proxy with no direct public backend access.
ROUTER_AUTH_FILE	./auth.json	Where keys are stored
ROUTER_CONTEXT_CONFIG	{"context_window":0,"model_limits":{}}	Persisted minimum context and confirmed per-model capacities. Prefer Settings → Context length; dashboard saves apply to new requests without a restart. See Context synchronization.
HERMES_INSTANCES_FILE	./instances.json	Where the dashboard’s Instances registry is stored. It may contain generated instance proxy keys and copied provider keys, so keep it private and never commit it.
CACHE_TTL_SECONDS	300	Response cache lifetime (0 disables). Entries are namespaced per API key, so different PROXY_API_KEYS do not share a cached answer
LOG_LEVEL	INFO	Logging verbosity
ROUTER_REQUEST_TIMEOUT_SECONDS	180	Cumulative upstream routing/failover budget; Requests timeouts cannot interrupt every DNS or trickling-read delay
SEMANTIC_EMBED_TIMEOUT_SECONDS	3	Maximum budget assigned to semantic-cache embedding work
REASONING_TOKEN_RESERVE	4096	Extra output budget added for reasoning models so hidden chain-of-thought doesn’t eat the answer (0 disables)
ROTATION_MODE	round-robin	How keys are picked within a provider (set in the dashboard or via hr mode) — round-robin or sequential

Context length and the VS Code extension

In Settings → Context length, choose Auto or a minimum model context. The minimum excludes smaller or unknown-capacity models and sets extension 0.9.10’s target budget. Auto uses confirmed capacities for extension discovery; other clients retain legacy Auto routing when capacities are unknown.

If discovery does not provide both context and output limits, enter the actual provider/account limits under Confirm model capacities. These are capacities, not limits you can increase beyond what the provider supports. The extension reserves output space and respects provider ceilings and application-key scopes.

The dashboard persists one validated JSON object in ROUTER_CONTEXT_CONFIG in the router’s environment file. context_window: 0 means Auto. model_limits maps provider::model to integer context_window and max_output_tokens values. Keep this file/volume during upgrades. Manual file edits require a process restart; dashboard saves do not. See Context synchronization for examples, upgrade requirements, and the application-key/admin API boundaries.

Advanced settings

Sensible defaults — most users never touch these.

Variable	Default	Purpose
MAX_REQUEST_BYTES	10485760 (10 MB)	Max request body size; larger requests get 413 (guards against memory exhaustion)
WORKER_THREADS	16	Waitress worker threads (concurrency). The HTTP connection pool scales with this
CACHE_MAX_SIZE	100	Max entries in the response cache (LRU eviction)
CACHE_PERSIST	0	If 1, mirror the cache to a SQLite file so it survives restarts (opt-in). The DB mirrors the in-memory LRU, so it stays bounded by CACHE_MAX_SIZE — raise that to persist more
CACHE_DB_PATH	./cache.db	SQLite file for the persistent cache. On read-only hosts (e.g. HF Spaces) point it at /tmp/cache.db
SEMANTIC_CACHE	0	If 1, also serve cached answers for similar prompts (needs an embedding provider; falls back to exact match otherwise)
SEMANTIC_CACHE_THRESHOLD	0.95	Cosine-similarity cutoff for a semantic hit (1.0 = identical; lower = looser matching)
FAST_ROUTE_THRESHOLD	0	If >0, requests under this many tokens prefer low-latency providers first (0 disables)
AUTO_DISCOVER_MODELS	0	If 1, refresh provider catalogs at startup; free chat imports are the default policy
AUTO_DISCOVER_FREE_MODELS	1	Add all confirmed free chat models; unknown prices are skipped
AUTO_DISCOVER_MODEL_LIMIT	8	Limit for legacy discovery only (AUTO_DISCOVER_FREE_MODELS=0)
{PROVIDER}_EXCLUDE_MODELS	—	Comma-separated model IDs to block for a provider (case-insensitive). Excluded models are stripped from config and discovery, e.g. OPENROUTER_EXCLUDE_MODELS=some/model:free
ROUTER_MODEL_ID	hermes-router	The model name clients send (the router maps it to each provider’s real model)
ROUTER_STATE_FILE	./router_state.json	Where provider ratings/capabilities are cached between restarts (use /tmp/... on read-only hosts like HF Spaces)
ROUTER_STATE_TTL_HOURS	24	How long the cached probe state is trusted before re-probing (0 = re-probe every start)
REQUEST_LOG_SIZE	500	Maximum in-memory request-metadata entries (0 disables the request log)
BREAKER_WINDOW	8	Recent outcomes the circuit breaker weighs per provider
BREAKER_MIN_SAMPLES	4	Minimum samples before the breaker can trip
BREAKER_ERROR_RATE	0.5	Health-failure fraction that trips the breaker
BREAKER_COOLDOWN	60	Seconds the breaker stays open before re-probing

Response-cache entries contain request material and generated responses. The in-memory cache retains them until expiry/eviction; CACHE_PERSIST=1 also writes them to CACHE_DB_PATH. Protect that file and disable caching where this retention is inappropriate.

Instance manager settings

The web dashboard’s Instances page stores its registry in HERMES_INSTANCES_FILE. A registry entry can be either:

• external — a name, base URL, and optional proxy key for a router you started elsewhere
• docker — a Docker image, host/container port mapping, generated or supplied proxy key, and env vars for a managed container

The file is written as JSON with 0600 permissions where possible and is listed in .gitignore. Treat it like auth.json: it can include secrets.

Variable	Default	Purpose
HERMES_INSTANCES_FILE	./instances.json	Registry file for monitored and Docker-managed instances
HERMES_INSTANCE_IMAGE	hermes-router:latest	Docker image used when the dashboard launches a managed instance
HERMES_INSTANCE_CONTAINER_PORT	8319	Port inside the managed container
HERMES_INSTANCE_DOCKER_PREFIX	hermes-router	Prefix for generated Docker container names

When creating a Docker instance, the dashboard can copy selected existing provider keys from the manager router into the child container. The browser only receives provider names and key counts; the backend reads the real keys server-side from auth.json and .env, then writes the matching container env vars, for example:

Provider	Env var copied into the instance
gemini	GEMINI_API_KEYS
openrouter	OPENROUTER_API_KEYS
github_models	GITHUB_MODELS_TOKENS
openai	OPENAI_API_KEYS
anthropic	ANTHROPIC_API_KEYS

Manual env vars entered in the dashboard’s Docker settings are merged with copied keys. If both set the same env var, the copied provider keys win for that provider.

Per-key budgets & rate limits

Give each PROXY_API_KEYS entry an upstream-usage ceiling. These env vars are global defaults; set per-key overrides in auth.json with hr limit set. 0 = unlimited (the default — no enforcement). Live usage shows in /v1/status and hr status.

Variable	Default	Purpose
PROXY_LIMIT_RPM	0	Requests/minute per key (rolling 60s window)
PROXY_LIMIT_REQ_DAY	0	Requests per UTC day, per key
PROXY_LIMIT_TOKENS_DAY	0	Tokens per UTC day, per key
PROXY_LIMIT_COST_DAY	0	Estimated USD cost per UTC day, per key (see Cost awareness below)

hr limit set sk-team-1 --rpm 60 --req-day 500 --tokens-day 100000 --cost-day 5   # per-key, written to auth.json
hr limit list                                                                    # show all
hr restart                                                                       # apply

Exceeding a limit returns 429 with a clear message and a Retry-After header. Per-key limits in auth.json look like:

{ "proxy_keys": { "sk-team-1": { "rpm": 60, "req_per_day": 500, "tokens_per_day": 100000, "cost_per_day": 5 } } }

Security boundary: every proxy key can also call the dashboard’s configuration-write endpoints. Limits and provider scopes restrict routing usage, but they do not create a read-only or non-admin role. Give keys only to trusted users/services and put an external auth layer in front if you need tenant isolation.

Cost / spend awareness

The router estimates spend from a built-in, manually maintained price table (USD per 1M tokens, input/output). Providers and subscription plans marked as zero-cost in that table show $0. Estimated cost appears per provider and per key in /v1/usage, /v1/status, hr status, the VS Code dashboard, and /metrics (hermes_router_cost_usd_total). USD is always the canonical figure.

Variable	Default	Purpose
COST_CURRENCY	USD	A second currency to also display (e.g. INR) — requires COST_FX_RATE
COST_FX_RATE	0	USD→COST_CURRENCY multiplier (e.g. 83); 0 shows USD only
MODEL_PRICES_FILE	(unset)	JSON file of price overrides — {"model-substr": [input, output]} (USD per 1M tokens) — merged over the built-in table

These are best-effort operational estimates, not provider invoices. Prices, promotions, tokenizers, and subscription terms drift; unknown models may show $0. Correct known prices with MODEL_PRICES_FILE.

Local model (Ollama / LM Studio / llama.cpp)

Set either of the first two to enable a local provider pointing at a model on your own machine. It’s keyless (cloud providers remain the fallback). See providers.md → Local models.

Variable	Default	Purpose
LOCAL_BASE_URL	http://localhost:11434/v1	Your local server’s OpenAI-compatible endpoint (LM Studio: :1234/v1)
LOCAL_MODEL	llama3.1	Local model id (comma-separate for multi-model failover)
LOCAL_API_KEY	local	Only if your local server actually requires a key
LOCAL_EMBED_MODEL	(unset)	Optional — also serve /v1/embeddings from the local server

Send model hermes-router:fast to prefer the local model for short/casual turns, with cloud fallback for heavier requests.

Per-provider model

Variable	Default	Purpose
ANTHROPIC_MODEL	claude-haiku-4-5-20251001	Model override (set via hr model set)
CODEX_MODEL	gpt-5.5	Codex (ChatGPT subscription) model — see providers.md
OPENAI_MODEL	gpt-4o-mini	Model override (set via hr model set)
GEMINI_MODEL	gemini-2.5-flash-lite	Model override (set via hr model set)
<PROVIDER>_MODEL	(varies)	Same pattern applies to all providers

Per-provider embeddings

Variable	Default	Purpose
GEMINI_EMBED_MODEL	gemini-embedding-001	Embedding model (empty disables this provider for /v1/embeddings)
<PROVIDER>_EMBED_MODEL	(gemini/mistral/cohere set)	Same pattern for embeddings; set empty to disable

Per-provider capability overrides

The router auto-probes each provider at startup, but you can force the result:

Variable	Default	Purpose
<PROVIDER>_SUPPORTS_TOOLS	(auto-probed)	Force tool-capability on/off (1/0)
<PROVIDER>_REASONING	(auto-probed)	Force reasoning-model on/off (1/0)
<PROVIDER>_SKIP_TOKENS_OVER	(per provider)	Skip this provider when an estimated request exceeds this many tokens (0 = never)
<PROVIDER>_MAX_OUTPUT_TOKENS	(per provider)	Clamp max_tokens down to this provider’s output ceiling (0 = no clamp)

Model overrides

Each provider has a default model that works out of the box. Switch models without editing files:

hr model list                              # see all providers and their active model
hr model set anthropic claude-sonnet-4-6   # upgrade Anthropic to Sonnet
hr model set openai gpt-4o                 # use full GPT-4o instead of mini
hr model set gemini gemini-2.5-pro         # switch Gemini to Pro
hr model reset anthropic                   # revert back to the default
hr restart                                 # apply changes

Overrides are stored as plain variables in .env (e.g. ANTHROPIC_MODEL=claude-sonnet-4-6) and active overrides are highlighted in hr model list.

Multiple models per provider

A provider can use several models — just give <PROVIDER>_MODEL a comma-separated list:

hr model set gemini gemini-2.5-flash-lite,gemini-2.5-flash,gemini-2.0-flash
hr restart

Some providers enforce model-specific limits, so each configured model can be another failover candidate. When the first model returns 429, the router fails over to the next model on the same key before cascading to the next provider. This does not bypass limits shared at the account, project, organization, token, or daily level. Each model is also a first-class routing candidate, scored on its own rating and capability — so the router can pick the right model in the list for each request (e.g. a stronger model for a hard or tool-using turn), not just fall over to it. Within equal cost/capability buckets, the router prefers known stronger model families, then falls back to your listed order.

Mixing model classes is fine. Tool-calling and reasoning are detected per model at startup, so you can safely list models of different classes (e.g. gemini-2.5-flash-lite,gemini-2.5-pro) — each is routed and gated on its own capability. Force a result per model with <PROVIDER>_<MODEL>_SUPPORTS_TOOLS / _REASONING (model id upper-cased, non-alphanumerics → _, e.g. GEMINI_GEMINI_2_5_PRO_SUPPORTS_TOOLS=1); the provider-wide <PROVIDER>_SUPPORTS_TOOLS / _REASONING still applies as the default for all its models.

Auto model discovery

Enable AUTO_DISCOVER_MODELS=1 to refresh eligible provider catalogs at startup. The dashboard automatic provider setup also enables discovery for that provider and runs it immediately in the background, without restarting the router.

AUTO_DISCOVER_FREE_MODELS=1 is the default discovery policy: import all confirmed free chat models, preserve explicit manual chat models, and keep model switches. <PROVIDER>_AUTO_DISCOVER_MODELS and <PROVIDER>_AUTO_DISCOVER_FREE_MODELS override the global choices. GROQ_DISCOVERY_FREE_TIER=1 is an operator assertion that the account uses Groq’s Free plan, not a detected property of its API key. See Providers for setup and provider links.

Unknown pricing is excluded. Reads use fixed provider endpoints, bounded time and size (4 MB / 5,000 models), no credential-bearing redirects and no inference probes. Unsupported pagination produces a visible error instead of silently importing an incomplete catalog. Failed reads keep the existing roster. Successful reads replace previous automatic additions while preserving manual chat selections and switches.

For the previous bounded discovery behavior, explicitly set AUTO_DISCOVER_FREE_MODELS=0. Only that legacy mode uses AUTO_DISCOVER_MODEL_LIMIT=8; it can add paid models. Leave free mode enabled for conservative automatic imports. Discovery is refreshed on save, explicit refresh, enabling discovery and startup, rather than continuously monitoring provider prices.

Per-provider exclude list

To permanently block specific model IDs for a provider — even when listed in {PROVIDER}_MODEL or re-added by auto-discovery — set {PROVIDER}_EXCLUDE_MODELS:

OPENROUTER_EXCLUDE_MODELS=some/unwanted-model:free
MISTRAL_EXCLUDE_MODELS=mistral-tiny
SAMBANOVA_EXCLUDE_MODELS=gemma-4-31B-it

Excluded models are matched case-insensitively (exact ID only, no globs). The filter applies both to your configured model list and to any extras appended by auto-discovery. If every model for a provider is excluded, the provider is shown in status but skipped for routing until at least one usable model is configured, and a warning is logged at startup.

Key rotation mode

When a provider holds several keys (or several accounts), ROTATION_MODE decides how the router picks among them. Set it in the dashboard’s Configuration panel or with:

hr mode                # show the current mode
hr mode round-robin    # default — spread requests evenly across all keys
hr mode sequential     # drain one key fully before moving to the next
hr restart             # apply the change

• round-robin (default) — every request goes to the next key in turn, so all keys share the load and deplete together. Best for spreading latency and load.
• sequential — one key is used until it hits its rate limit, then the router moves to the next, and so on. Later keys stay untouched in reserve — useful when you want to ration accounts after a quota reset instead of burning them all at once. Keys are drained in the order they appear in auth.json.

Either way, failover, per-key cooldowns, and the circuit breaker keep working — the mode only changes which ready key is preferred next. The active mode shows in hr status and at /v1/status.