Configuration

The apiserver is configured with a YAML file passed via --config, with individual command-line flags, or with environment variables. A complete annotated example lives at configs/apiserver/config.sample.yaml.

go run ./cmd/apiserver/main.go --config ./configs/apiserver/dev.yaml

Precedence

Every YAML key has an equivalent dotted flag and environment variable. Command-line flags override the config file. For example management.log.level can be set as:

--management.log.level=debug          # flag
MANAGEMENT_LOG_LEVEL=debug            # environment variable

External secrets

Keep plaintext credentials in YAML only for local development. In production, set a nonempty environment variable or its _FILE counterpart pointing to a readable secret file. Names use uppercase with dots replaced by underscores; camel-case words are not separated (for example clientSecret → CLIENTSECRET).

AUTH_JWT_SECRET_FILE=/run/secrets/jwtSecret \
AUTH_BASIC_PEPPER_FILE=/run/secrets/basicPepper \
DATABASE_ENDPOINTS_FILE=/run/secrets/databaseURI \
  /apiserver --config /etc/opampcommander/config.yaml

Precedence for these fields is explicit flag → nonempty environment value → file → YAML → default. An empty environment value or _FILE path is treated as unset; an explicitly empty flag still wins. Higher-priority values skip reading the file. A selected file that is unreadable or empty stops startup, without falling back to YAML. Files contain the literal value, with one trailing LF or CRLF removed; spaces are preserved. DATABASE_ENDPOINTS_FILE contains one complete MongoDB URI, preserving commas in replica-set host lists. TLS private keys already use tls.keyFile / management.tls.keyFile directly.

Secrets are read once at startup; restart the server after rotation. Keep the basic-auth pepper stable: changing it invalidates stored password hashes. Restrict file permissions to the server user and keep credentials out of ConfigMaps, Git, and command-line arguments. For Helm-mounted Secrets, see the chart example.

apiserver config view masks secret fields, including database endpoints. opampctl config view masks user passwords and bearer tokens without changing the saved configuration. Both commands accept --show-secrets for intentional plaintext output; do not send that output to shared logs. File source comments list only the environment variable names, never file contents. MongoDB URI validation errors omit driver details because those can contain credentials; check database.endpoints if startup reports an invalid MongoDB configuration.

Server

address: localhost:8080    # REST API + OpAMP WebSocket endpoint
serverId: ""               # defaults to hostname; also settable via SERVER_ID
serviceName: opampcommander

Graceful shutdown

shutdown:
  drainWindow: 10s
  timeout: 30s

On shutdown, /readyz returns 503 and the OpAMP endpoint rejects new connections. Existing WebSockets receive a Going Away (1001) close frame at randomized times within drainWindow. The server waits for close handshakes, then shuts down the HTTP listener and drains REST requests. At the total timeout, any remaining WebSockets are force-closed. Zero values select the defaults above; the drain window must be nonnegative and shorter than the timeout. The same settings are available as --shutdown.drainWindow, --shutdown.timeout, SHUTDOWN_DRAINWINDOW, and SHUTDOWN_TIMEOUT.

Set the pod’s terminationGracePeriodSeconds longer than shutdown.timeout so Kubernetes leaves enough time for this shutdown sequence.

Listener TLS

Set tls.certFile and tls.keyFile to enable HTTPS for the entire REST API and WSS at /api/v1/opamp. Both paths are required; invalid or unreadable credentials fail startup. Clients must trust the CA that issued the server certificate and connect using a name present in its subject alternative names.

tls:
  certFile: /etc/opampcommander/tls/server.crt
  keyFile: /etc/opampcommander/tls/server.key
  minVersion: "1.2"
  cipherSuites: []
management:
  tls:
    certFile: /etc/opampcommander/tls/management.crt
    keyFile: /etc/opampcommander/tls/management.key

certFile contains the PEM leaf certificate followed by any intermediate certificates; keyFile contains its PEM private key. Protect the key with file permissions readable only by the server. minVersion accepts "1.2" (default) or "1.3"; older versions are rejected. cipherSuites optionally restricts TLS 1.2 to named secure suites from Go’s tls.CipherSuites, for example TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256. Empty uses Go defaults. Go controls preference order and TLS 1.3 suites; TLS 1.3 suite names cannot be configured.

The management listener is configured independently and serves health checks, metrics and pprof. Its optional management.tls.caFile requires a trusted client certificate on every management route. Update health probes and Prometheus scrapers to use HTTPS and the necessary client credentials. Keep this listener private even when using TLS.

All new settings have dotted flags and environment variables, for example --tls.certFile, TLS_CERTFILE, and MANAGEMENT_TLS_KEYFILE.

Without certificate paths, a listener stays plaintext for local development or ingress TLS termination. The server logs a startup warning naming the plaintext OpAMP WebSocket endpoint, whose remote configuration can contain credentials, and a separate management warning. Do not expose either plaintext listener to an untrusted network.

Server certificate rotation

Replace the server certificate and key, then send SIGHUP to the apiserver process:

kill -HUP <apiserver-pid>

Both configured listeners reload their own server certificate/key pairs. New TLS handshakes use the replacement; existing WebSocket and HTTP connections continue uninterrupted. Missing, malformed, or mismatched replacements log an error and retain the last valid pair. Publish both files before signaling, preferably by switching a directory symlink to a complete pair. Repeat for every server process. Settings, paths and client CA bundles are loaded at startup and require a restart to change.

OpAMP mutual TLS

Add tls.caFile to trust a PEM bundle of agent client CAs. Only /api/v1/opamp requires an agent client certificate; REST clients may connect without one. Each agent certificate’s CN must equal its instance UID.

tls:
  certFile: /etc/opampcommander/tls/server.crt
  keyFile: /etc/opampcommander/tls/server.key
  caFile: /etc/opampcommander/tls/agent-ca-bundle.pem

Existing opampTLS configurations remain supported with all three paths required. They also gain SIGHUP server-certificate reload. Do not configure certificate paths under both tls and opampTLS; ambiguous settings fail startup.

Issue agent client certificates through your external CA or PKI tooling; APIServer does not issue certificates or load a CA private key. Register an issued client certificate and its private key as a Certificate resource, then offer it to a connected agent through its single-agent group’s OpAMP connection settings. For bootstrap, install the certificate and key on the agent before its first connection. Grant read access to Certificate resources only to operators allowed to retrieve agent private keys.

The Certificate’s caCert field is the CA the agent trusts for the server certificate. It can differ from the CA that signs agent client certificates.

The rotation lifecycle is: externally issue and register a Certificate, offer it through ConnectionSettings, then confirm reconnection and an applied connection settings status on the Agent resource. This requires a client that accepts OpAMP connection settings and reports their application status; Supervisor 0.163.0 does not qualify. See the connection-settings support policy. Once the replacement certificate is active, messages using the old certificate are rejected and the previous connection on the same server node is closed. A previous connection on another node is rejected on its next message. This is an OpAMP authorization change; revocation at the issuing CA remains an external PKI operation.

To rotate an agent-client CA, put the old and new CA certificates in the same PEM file, one BEGIN CERTIFICATE block after the other. Restart every apiserver instance so both CAs are trusted, switch agents to certificates issued by the new CA, then remove the old CA from the bundle and restart the instances again. Client CA bundles are read only at startup; SIGHUP reloads only the server certificate/key and does not change the CAs trusted by a running instance. caFile does not control which CA an agent trusts for the server certificate.

OpAMP transport

opamp:
  maxMessageSize: 0
  enableCompression: false

maxMessageSize limits sent and received OpAMP messages after decompression: HTTP uses the full request/response body; WebSocket uses the full OpAMP message. 0 uses the opamp-go default (64 MiB), any negative value disables the limit, and a positive value sets the limit in bytes. Disabling the limit permits unbounded message memory use; prefer a finite limit suitable for your agents.

Compression defaults to false; set enableCompression: true to enable transport compression (WebSocket also requires client negotiation). Both settings require a restart. Flags are --opamp.maxMessageSize and --opamp.enableCompression; environment variables are OPAMP_MAXMESSAGESIZE and OPAMP_ENABLECOMPRESSION. For example, OPAMP_MAXMESSAGESIZE=8388608 sets an 8 MiB limit.

Database

database:
  type: "mongodb"          # "mongodb" or "inmemory" (inmemory for local/dev only)
  endpoints:
    - "mongodb://<user>:<password>@localhost:27017"
  connectTimeout: 10s
  databaseName: "opampcommander"
  ddlAuto: true            # create indexes/schema on startup

inmemory keeps no data across restarts and is intended for development and tests.

Sharded cluster (MongoDB)

To scale horizontally beyond a single shard, point endpoints at your mongos routers and turn on sharding-aware schema management. This is opt-in and requires ddlAuto: true.

database:
  type: mongodb
  endpoints:
    - "mongodb://<user>:<password>@mongos-0:27017,mongos-1:27017/?replicaSet="
  databaseName: opampcommander
  ddlAuto: true
  sharding:
    enabled: true          # enableSharding + shardCollection on startup

When sharding.enabled is set, startup additionally runs enableSharding on the database and shardCollection for the collections that grow with fleet size. The operation is idempotent — restarts against an already-sharded cluster are a no-op.

The shard-key plan is built into the server (it is tied to the collections’ unique indexes and query patterns, so it is not operator-configurable):

All other collections (config-/cluster-scale: agentgroups, agentpackages, agentremoteconfigs, certificates, endpoints, namespaces, servers, serverheartbeats, users, and the RBAC collections) stay on the primary shard — sharding them would add routing cost with no benefit. They are still created explicitly at startup so they land deterministically. hosts and containers are sharding candidates for very large fleets; they are created but not yet sharded.

Events (single-node vs. multi-node)

event:
  enabled: false           # false = standalone (single instance)
  type: "inmemory"         # "inmemory" for standalone, "kafka" for distributed
  kafka:
    brokers:
      - "localhost:9092"
    topic: "prod.opampcommander.events"
    sendTimeout: 2s        # enqueue and individual producer/socket operation timeout
    retryBackoff: 100ms    # delay between attempts
    retryAttempts: 2       # includes the initial attempt; 1 disables retries

When running multiple apiserver instances, set enabled: true and type: kafka so a management request received by one instance can be delivered to an agent connected to another. See the protocol overview for the coordination flow.

The Kafka delivery settings above are the defaults. Omitted or zero values use the defaults; negative values are rejected at startup. They can also be set through --event.kafka.<setting> flags or environment variables such as EVENT_KAFKA_SENDTIMEOUT=3s. sendTimeout also bounds producer/socket operations.

Kafka events are best-effort notifications that accelerate applying saved changes. There is no application replay queue; Sarama owns its normal producer buffer and retries. An agent update succeeds once its desired state is saved; notification failure does not fail the API request. Agents also fetch desired state on heartbeat responses. If a cache-invalidation event is lost, the fixed cache TTL bounds how long a peer can serve its cached copy.

Sarama handles retries with the configured backoff. A successful send means the notification was submitted to the producer; delivery failures are logged asynchronously. sendTimeout limits enqueue waits and individual producer/socket operations, not the entire background retry cycle. Graceful shutdown asks the producer to flush pending messages while draining errors, with a bounded shutdown wait.

Agent liveness

Agent liveness — connected, lastReportedAt, sequenceNum — churns on every heartbeat but is worthless once stale: the next message from the agent rebuilds it. The server absorbs those updates in a fast tier and writes them through to the database on a slow cadence, so a large fleet stops paying a database write per agent per heartbeat.

liveness:
  flushInterval: 30s       # write-behind cadence
  flushStaleAfter: 30s     # how far behind a document must fall to be claimed
  flushBatchSize: 2000
  # persistThrottle: 10s   # leave unset unless you have a reason
  redis:
    enabled: false         # opt-in; absent or false is the default behaviour
    endpoints:
      - "redis:6379"
    dialTimeout: 2s
    commandTimeout: 200ms
    ttl: 120s

What the settings mean

flushInterval and flushStaleAfter bound how stale the stored agent document may get, and it is their sum that matters: a document ages by up to flushInterval + flushStaleAfter, because the flush only claims documents already flushStaleAfter behind and only looks every flushInterval. That sum has to stay inside the 60s budget the 90s connection-staleness window allows, and the server refuses to start otherwise.

This is not just about surviving an outage. “Connected” is evaluated inside the database in two places no read-side overlay can reach — the connected-agent list filter and the agent-group connected counts — and both read the stored timestamp. Let the document drift past the window and both start reporting live agents as disconnected while their WebSockets are fine.

persistThrottle is the minimum interval between database writes on the message path for an agent whose only change is that it is still alive. Leave it unset: 10s with no shared fast tier (where it is the only thing keeping the document current), and the 60s budget with one (where the write-behind flush owns the routine write path). It is capped at the budget either way, for the same reason.

ttl must exceed the 90s staleness window, or a live agent’s record would expire while the agent is still considered connected. The server refuses to start otherwise.

commandTimeout is short on purpose. This is a fast path: a slow Redis must degrade to the database rather than hold up agent message processing.

Redis is optional, always

Redis is an accelerator and never a requirement. Absent, misconfigured, or down, the server keeps working on node-local state and the database — just with more database writes.

• A Redis that is unreachable at startup does not stop the server from starting.
• A Redis that fails while running trips a circuit breaker after a few consecutive failures. Calls route to node-local state, the server forces a write-behind flush so the database is current before reads start relying on it, and a probe every 30s routes back automatically once Redis returns. No restart, no config change.
• The outage shows up as degraded in /healthz, with the status code left at 200: losing an optional accelerator must never be the reason a process is restarted. /readyz is unaffected.
• Redis is ignored entirely in standalone mode (database.type: inmemory), with a warning in the log. A standalone server keeps its whole state in process, so a shared tier would only add a network hop to reach data it already holds.

One endpoint addresses a single server, several address a cluster, and adding masterName selects Sentinel.

What it actually saves

The saving is not primarily fewer writes — the staleness window puts a floor under how rarely the stored document can be refreshed. It is much cheaper writes: a four-field $set instead of a full document rewrite, with no resource-version bump and so no optimistic-concurrency churn against the reconcile loop and API writes.

Measured with make bench-liveness, per heartbeat per agent:

At the standard 30s OpAMP heartbeat, every full document rewrite becomes a narrow $set — the same write rate, at a fraction of the cost, and with the version churn gone. A chattier fleet wins on rate too: at a 5s heartbeat the write rate drops about six-fold on top of that.

Note the middle rows: a throttle shorter than the heartbeat interval buys nothing, because it has always elapsed by the time the next heartbeat lands. That is why the throttle alone was never the answer.

Metrics

absorbed minus written is the database writes the fast tier saved.

Bootstrap (initial manifests)

On startup the server reconciles a directory of manifest YAML files into persistence (declarative, full overwrite). The container image ships defaults at /etc/opampcommander/initial (also exposed via the BOOTSTRAP_DIR env var).

bootstrap:
  dir: /etc/opampcommander/initial
  remoteConfigSchemaLoad: latest   # latest | all | none
  defaultNamespace: default   # namespace for agents without a service.namespace
  defaultRole: default        # role auto-granted to every user

Setting bootstrap.dir empty disables bootstrapping.

RemoteConfigSchema library

The image also ships a pre-built library of RemoteConfigSchema manifests under <bootstrap.dir>/remoteconfigschema: one per OTel Collector release and distribution, listing the components that release ships, the signals each one handles, its stability, and the settings it accepts. On startup these are seeded per bootstrap.remoteConfigSchemaLoad:

• latest (default) — seed only the newest version of each distribution (otelcol, otelcol-contrib, otelcol-k8s, otelcol-otlp)
• all — seed every version in the library
• none — seed nothing (the files remain on disk for opt-in use)

Point bootstrap.remoteConfigSchemaDir elsewhere to use a different library.

The library is rendered from the otelcol-config-schemas registry, whose files are generated by tools/schemagen in otelcol-config-lint from each release’s upstream metadata.yaml, its components’ Config structs, and the config.schema.yaml the collector publishes from v0.150.0 on. The linter reads the same registry, so a config checked locally with otelcol-config-lint is checked against the same catalog the server holds.

Regenerate the library with make gen-remoteconfigschema, or pull a single release yourself:

opampctl generate remoteconfigschema --distribution contrib | opampctl create -f -
opampctl generate remoteconfigschema --distribution core --version v0.150.0 > schema.yaml

Releases the registry does not cover are handled by the collector itself: point --binary-path at any distribution, including a custom one, and its components output becomes a schema. Such a schema lists the components the binary has but not the settings they accept, so configs targeting it are checked for component existence and signal support only.

opampctl generate remoteconfigschema --binary-path ./otelcol-custom > schema.yaml

Management (observability)

The management server runs on a separate address and hosts health checks, metrics, pprof, logging, and tracing configuration.

management:
  address: localhost:9090
  metric:
    enabled: true
    type: prometheus              # "prometheus" or "opentelemetry"
    prometheus:
      path: /metrics
    opentelemetry:                # used when type is opentelemetry
      endpoint: "localhost:4317"
  log:
    level: "info"                 # debug, info, warn, error
    format: "json"                # json or text
  trace:
    enabled: false
    endpoint: "localhost:4317"
    protocol: grpc                # grpc, http/protobuf, http/json
    sampler: always               # always, never, probability

Health checks are served at GET /healthz and GET /readyz.

Both endpoints return a JSON object mapping affected dependency names to reasons (an empty object when there are no problems). Their dependency semantics are:

Kafka broker reachability is checked with a metadata request, without producing events or creating topics. Results, including failures, are cached for five seconds per server; concurrent polls share one check. Socket operations are limited to 200 ms and metadata attempts have a 500 ms budget (an in-flight socket operation can finish after that budget). Recovery appears on the first health poll after the cache expires. Monitor the /healthz JSON body to detect degraded cross-server command delivery: Kubernetes probes intentionally stay successful so a broker outage does not drain or restart every replica.

Authentication

OpAMP Commander supports OAuth2 (GitHub), basic auth (with hashed passwords), and a manual bearer-token mode used by the CLI.

auth:
  enabled: true
  admin:
    username: "admin"
    password: "admin_password"
    email: "admin@admin"
  basic:
    # Server-side secret mixed into every basic-auth password hash. Set a long,
    # random, stable value. Empty disables DB-backed basic-auth users.
    pepper: ""
  jwt:
    issuer: "opampcommander"
    expire: 30m                   # access token lifetime
    refreshExpire: 168h           # refresh token lifetime (0 disables refresh)
    secret: "your_jwt_secret"
    audience:
      - "opampcommander"
  type: "oauth2"
  oauth2:
    provider: github
    clientId: "your_client_id"
    clientSecret: "your_client_secret"
    redirectUri: "http://localhost:8080/auth/callback"
    # Extra hosts the authcode endpoint accepts as redirect targets, on top of the
    # always-allowed loopback hosts (127.0.0.1, ::1, localhost). Add your web UI host.
    allowedRedirectHosts:
      - opampcommander.minuk.dev
    state:                        # CSRF protection for the OAuth2 flow
      mode: jwt
      jwt:
        issuer: "opampcommander"
        expire: 5m
        secret: "your_jwt_secret"
        audience:
          - "opampcommander"

With auth.enabled: false, authentication is bypassed — only suitable for local development.

Command-line flags

Every option above has a flag. A few common ones:

Run apiserver --help for the complete list.