# SPEC — Telemetry Cleanup (Priority 2, final piece)

**Status**: Plan phase in progress — revised twice. First: an independent Opus 5 Spec Revision pass found two factual errors in the original "empirical" audit (one of them undermining the spec's largest Feature Design) plus several real feasibility gaps. Second: a Plan-phase investigation (triggered by user follow-up questions on the tailnet-auth gap, the monitoring-workload verification gap, and whether MediaMTX's own custom image could do better than attribute-based trace correlation) resolved all three with concrete, source-grounded designs — see Feature Design 2c (new) and the updated Feature Design 3b. See "Revision Changelog" at the end for exactly what changed and why, across both passes.

**Written against**: `plevion-k8s` `main` @ `dbe1c31` (Phase 2 core merge), `auth`/ `orchestrator`/`edge-controller`/`sink` `main` tips as of 2026-08-01, `authmtx` `main` @ `6213123`, `mediamtx` `main` @ `4119e26`, `kong` `main` @ `c9e2020`, `frontend-admin`/ `frontend-landing` `main` tips as of 2026-08-01.

## Objective

Close out the last open piece of `plevion-docs/ROADMAP.md` Priority 2. `ROADMAP.md`'s telemetry section has four bullets; this spec covers all four, though bullets 1 and 4 (log readability / collector pipeline verbosity) turn out to be **directly coupled** to bullet 3 (trace correlation) rather than separable — see Feature Design 6's "log noise vs. trace correlation" subsection, forced into scope by Spec Revision finding 14. Concretely: deploy the monitoring stack at all (it currently never is), close the handful of real per-service instrumentation gaps that remain once the (mostly already-wired) Python services are correctly audited, get both Go services (`mediamtx`, `authmtx`) into the telemetry picture with an honest design for what "trace correlation" actually means for each, enable Kong's tracing (the single entry point for all traffic), and standardize resource-attribute tagging so results are queryable by service/version/environment — without breaking the log-to-trace correlation Grafana already depends on.

## Correcting the original framing — read this before anything else

This effort was originally framed as "add a way to start the cluster **without** the monitoring workload," on the assumption that monitoring is deployed by default today and needs an opt-out. **That assumption is wrong, and it's the root cause of nearly everything else in this spec, not just a convenience gap.**

- `ansible/roles/infrastructure/tasks/main.yml` applies `infrastructure/overlays/{env}` only (MetalLB, cert-manager, Tailscale operator).
- `ansible/roles/services/tasks/main.yml` applies `services/overlays/{env}[-with- frontend]` only.
- **No role applies `monitoring/overlays/{env}` or `environments/local/` as a whole.** Confirmed exhaustively: `ansible/roles/` contains exactly `cluster_bootstrap, frontend_docker, infrastructure, migrations, postflight, preflight, secrets, services` — none references `monitoring/`.

**This means the OTel Collector is never running on `local`, ever, regardless of any other change in this spec.** Every currently-instrumented service (see the corrected audit below — this is most of them) is exporting into a void: `OTEL_EXPORTER_OTLP_ENDPOINT: http://$(HOST_IP):4317` with nothing listening on that port. **Feature Design 1 is not one item among several — it's the blocking prerequisite for every other Feature Design in this spec to be verifiable at all.**

**Confirmed with the user**: follow the existing `--with-frontend` precedent exactly (opt-in, off by default) — new flag is `--with-monitoring`. See Feature Design 1 for a real consequence of this default that the original draft missed.

## Current-state audit (corrected — the original draft got this wrong for 4 of 7 services)

The original draft claimed `auth`/`orchestrator`/`sink`/`edge-controller` all have "OTel packages listed, zero runtime wiring." **This was true only for `edge-controller`.** The other three are wired via `command: ["opentelemetry-instrument", ...]` overrides in their k8s Deployments/Jobs — a location the original audit didn't check (it looked at each repo's own `docker/` directory, which has no `CMD`/`ENTRYPOINT` at all; the runtime command lives entirely in `plevion-k8s/services/*/base/*.yml`).

| Service / workload | Wired? | `OTEL_SERVICE_NAME` set? | Gap |
|---|---|---|---|
| `auth` (`deployment.yml`) | ✅ `opentelemetry-instrument gunicorn` | ❌ | Add `OTEL_SERVICE_NAME` |
| `auth` events gateway (`gateways/events/deployment.yml`) | ✅ | ❌ | Add `OTEL_SERVICE_NAME` |
| `orchestrator` (`deployment.yml`) | ✅ | ❌ | Add `OTEL_SERVICE_NAME` |
| `orchestrator-worker` | ✅ `opentelemetry-instrument celery` | ✅ | none |
| `cqrs-worker`, `monitor-controllers-worker` | ✅ | ✅ | none |
| `reap-connections-cron`, `check-components-updates-cron` | ❌ `command: ["python"]` | n/a | Add wrapper |
| `sink` (`deployment.yml`) | ✅ | ✅ | none |
| `sink` broadcast gateway (`gateways/broadcast/deployment.yml`) | ❌ `command: ["gunicorn"]` | n/a | Add wrapper |
| `sink-worker`, `sink-capturer-worker` | ✅ `opentelemetry-instrument celery` | ✅ | none |
| `sink-cqrs-worker` | ⚠️ `command: ["uv","run","python","manage.py","cqrs_consume"]` | ✅ | Wrapper must go *inside* `uv run`, not before it |
| `edge-controller` (`deployment.yml`) | ❌ `command: ["celery"]` | n/a | **Zero OTel deps anywhere in the repo at all** (`pyproject.toml` has no `opentelemetry-*` package, `CLAUDE.md` has zero mentions) — needs the full dependency set added, not just a wrapper. Real cost: 7 new packages on a resource-constrained RPi5 in the real-device deployment mode (Feature Design 2b) — a genuine tradeoff, not free. |
| `frontend-landing` | ✅ real `NodeSDK` in `instrumentation.node.ts` | ✅ (`plevion-landing`) | Only `service.name` set — no `service.version`/`deployment.environment` |
| `frontend-admin` | ✅ same | ✅ (`plevion-admin`) | Same gap |
| `mediamtx` | N/A (upstream binary) | — | See Feature Design 3a |
| `authmtx` | ❌ zero telemetry | — | See Feature Design 4. `services/mediamtx/base/authmtx.yml` already has a `service: {name, version, environment}` block — but `version: "latest"`/`environment: "local"` are **hardcoded literals**, not read from env. |
| `kong` | ❌ zero telemetry configured | — | See Feature Design 5 |

**Even the fully-wired services (most of `auth`/`orchestrator`/`sink`) emit nothing useful today** — not because they're unwired, but because Feature Design 1 means there's nothing listening on the other end. Fixing FD1 first, then verifying what already flows, then closing the small residual per-workload gap list above, is the real shape of this work — not "instrument everything from scratch."

## Confirmed Scope Decisions (resolved with the user before this revision)

1. **Monitoring deployment**: new opt-in `--with-monitoring` flag. Off by default, matching `--with-frontend`'s precedent exactly. **Revised**: see Feature Design 1 for a real consequence of "off by default" the original draft didn't account for (every wired service's OTLP exporter connection-refused-loops against a collector that doesn't exist, in the default case).
2. **~~Python services get real instrumentation, not just an audit~~ — revised.** Most of `auth`/`orchestrator`/`sink` are already wired (see corrected audit above). The real scope is the specific gap list in that table, plus `edge-controller`, which genuinely does need real instrumentation added from zero.
3. **`edge-controller` has two real deployment modes, both in scope** — unchanged from the original framing, but see Feature Design 2b for corrected mechanics: the in-cluster case is not "trivial" (it needs the same wrapper + env vars as every other service, currently has neither), and the tailnet case needs a new Service resource that doesn't exist yet, not just a flag.
4. **`mediamtx` needs trace correlation, not just metrics — revised twice now: once to attribute-based correlation, once further to genuine shared-trace correlation.** MediaMTX's `authHTTPAddress` auth-decision call genuinely has zero header/context customization surface — confirmed by reading MediaMTX's own source at the pinned tag (`internal/auth/manager.go`'s `authenticateHTTP` does a hardcoded `httpClient.Post` with a plain `&http.Transport{}`), so there is still no way to carry an inbound `traceparent` from the triggering RTSP/WebRTC/SRT connection (which has no HTTP headers to propagate from in the first place) — that part of the original correction stands.

   **But a second Plan-phase investigation (triggered by the question "can the custom `mediamtx` image itself inject tracing?") found something better than root-span-plus- attributes**: MediaMTX already generates **one stable UUID per connection**, reused unchanged as the `/auth` call's `id` field, `$MTX_SOURCE_ID` in the `runOnReady`/ `runOnNotReady` hook environment, and the `APISourceDescribe` ID — confirmed by tracing `c.uuid` through MediaMTX's own `internal/servers/rtmp/conn.go` and `internal/hooks/on_ready.go` source at the pinned tag, not assumed. `authmtx`'s `handler/auth.go` already decodes this ID from `/auth`'s JSON body today; it's simply unused. Passing the same ID to the `runOnReady`/`runOnNotReady` hooks too (a one-line `mediamtx.yml` query-param addition — no image/binary change needed at all, despite the question assuming a custom wrapper binary would be required) lets `authmtx` derive a **deterministic trace ID** from that UUID (`uuid.UUID` and OTel's `trace.TraceID` are both `[16]byte` — a direct reinterpretation, no hashing) for all three hook calls. Result: `auth`/`ready`/`notready` for one MediaMTX connection become **three spans sharing one real, queryable trace ID** in Tempo — genuine same-trace correlation for that connection's full lifecycle, not attribute-search correlation across three disconnected traces. The synthetic parent isn't a real recorded span, so Tempo will likely render these as top-level spans within that one trace rather than a clean parent→child tree (confirm exact rendering empirically during Build) — cosmetically imperfect, but a real, substantive upgrade. See Feature Design 3b for the full mechanism and file-level detail.

   **What still isn't achievable, unchanged from the first correction**: this shared trace still doesn't nest under `sink`/`orchestrator`'s own distributed trace for the same stream (no inbound trace context exists to link them) — cross-referencing those still happens by looking up the shared object ID in Tempo, not by trace nesting.
5. **`authmtx` gets real OTel Go SDK instrumentation** — unchanged, but see Feature Design 4 for an honest dependency-cost accounting (adding the gRPC-based OTLP exporter pulls in dozens of transitive modules, not "small and well-scoped" as originally stated — mitigated by using the HTTP exporter instead, see below).

## Feature Design 1 — `--with-monitoring` flag + new `monitoring` ansible role

**Files**: `ansible/bootstrap.sh`, `ansible/site.yml`, new `ansible/roles/monitoring/tasks/main.yml`, `monitoring/overlays/local/values/otel.yml` (new Service resource, see below), `monitoring/overlays/dev/` (new — see dev-guard below).

Mirror `with_frontend` exactly:
- `bootstrap.sh`: new `--with-monitoring` flag, `WITH_MONITORING="false"` default, passed through as `-e "with_monitoring=${WITH_MONITORING}"`.
- `site.yml`: new `with_monitoring: false` var default; new `monitoring` role added to the `roles:` list, `when: with_monitoring | bool`, placed after `infrastructure` and before `migrations`/`services` (no race found against either — `monitoring` targets its own namespace and nothing in `migrations`/`services`/`postflight` reads from it at apply time).
- New role applies `monitoring/overlays/{env}` and waits for the OTel Collector DaemonSet to have at least one `Ready` pod **and** confirms the readiness check targets the actual path services use (`hostIP:4317`, not just "pod is Running" — see below for why this distinction matters) before returning.

**`--env=dev` guard, new in this revision**: `monitoring/overlays/` currently has only a `local/` subdirectory. `site.yml`'s `Validate env` assert accepts `env in ['local', 'dev']`, so `--with-monitoring --env=dev` is a reachable flag combination that would fail on a missing kustomize target. The `monitoring` role must either assert `monitoring/overlays/{env}` exists before attempting to build it (clear failure message, not a raw kustomize error) or `dev` parity needs to be built as part of this phase — decide in Plan; at minimum this is a guarded failure mode, not silently ignorable the way the original draft's Open Question framed it.

**In-cluster reachability mechanism — corrected, new in this revision**: every currently-wired service reaches the collector via `$(HOST_IP):4317` (a `fieldRef: status.hostIP` + the DaemonSet's `hostPort: 4317`), **not** a Kubernetes Service. This already works today for any pod on a node running a collector instance and needs no new Service for the in-cluster case. **A new Service *is* needed, but only for Feature Design 2b's tailnet-exposure case** (an edge device has no concept of "the node's host IP" the way an in-cluster pod does) — see FD2b for why this must be a manually-authored Service (not the chart's own `service.enabled` toggle, which defaults to `internalTrafficPolicy: Local` in DaemonSet mode and would only route to whichever collector replica happens to share a node with the Tailscale proxy pod).

**Default-off consequence — new in this revision**: with monitoring off by default (the confirmed decision), every already-wired service's OTLP exporter will connection-refuse-loop against a nonexistent collector on every plain `bootstrap.sh` run. OTel SDK exporters are designed to fail this way non-fatally (retry-with-backoff in a background thread, not a startup-blocking or request-blocking failure) — but this is worth confirming empirically in Plan/Build rather than assumed, and is plausibly a real contributor to the log-verbosity complaint this same ROADMAP item flags (a connection-refused retry loop is exactly the kind of noise a developer would want stripped, and stripping it without understanding the cause could look like "fixing" something that's actually inherent to the opt-in design). Document this explicitly wherever exporter behavior is discussed with developers, rather than leaving it as a silent surprise.

**Version-pin boundary, new in this revision**: `otel.yml` pins `image.tag: "0.105.0"` while presumably newer chart versions exist — this pin is **load-bearing**, not stale by oversight: the collector's logs pipeline uses the `loki` exporter, which upstream deprecated in mid-2024 (scheduled for removal, replacement is `otlphttp`). Do not "helpfully" bump this image tag as a drive-by while touching this file for the new Service resource or the `prometheus` receiver (Feature Design 3a) — that would silently break the entire logs pipeline. Call this out explicitly as a **Never** in Boundaries.

**Transition-away case** (re-running without `--with-monitoring` after previously running with it): whether to tear down monitoring or leave existing Loki/Tempo data in place remains an **Open Question** — not resolved by this spec.

## Feature Design 2a — Close the remaining Python instrumentation gaps

**Files**: per the corrected audit table above — `services/orchestrator/base/crons/ {reap-connections,check-components-updates}-cron.yml`, `services/sink/base/gateways/ broadcast/deployment.yml` (add `opentelemetry-instrument` wrapper to each); `services/ sink/base/workers/sink-cqrs-worker.yml` (wrapper must wrap *inside* the `uv run` invocation: `uv run opentelemetry-instrument python manage.py cqrs_consume`, not before `uv run`, or the instrumentation never actually runs under `uv`'s resolved environment); `services/auth/base/deployment.yml`, `services/auth/base/gateways/ events/deployment.yml`, `services/orchestrator/base/deployment.yml` (add `OTEL_SERVICE_NAME`, currently missing — these three currently report as `unknown_service` if they ever did export successfully).

This is a small, enumerable list — not "instrument every Python service from scratch." Verify `opentelemetry-bootstrap` (the distro's auto-instrumentation-detection step) has actually run for each repo's dependency set before assuming the already-wired `opentelemetry-instrument` calls have anything registered to instrument — check each repo's build step, don't assume.

## Feature Design 2b — `edge-controller`'s two deployment modes

**Files**: `edge-controller`'s `pyproject.toml` (new OTel deps — the full set, since none exist today), `edge-controller`'s Dockerfile/entrypoint (wrapper), `plevion-k8s/ services/edge-controller/base/deployment.yml` (currently has neither the wrapper nor any `OTEL_*` env vars — **not the "trivial" case the original draft claimed**, this needs the same treatment as every other service in Feature Design 2a, from zero), plus:

- `orchestrator/src/apps/components/ansible/templates/controllers.py` — the `EDGE_CONTROLLER_TMPL`'s `environment:` block already has exactly the right precedent (`SERVICE_VERSION: {{ edge_controller_version }}`, confirmed by direct read), so add `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_SERVICE_NAME` there the same way. **Also** add the `opentelemetry-instrument` wrapper to this same template's `command:` (currently `celery -A settings worker -l info ...` — env vars alone do nothing without the wrapper, same as every gap in Feature Design 2a). Name the new Ansible-rendering context value explicitly in Plan — the call site (`orchestrator/src/apps/components/ansible/playbooks/components.py`) sources everything else from `settings.*` (e.g. `settings.API_URL`), so the new OTLP-endpoint value should follow that same pattern (e.g. `settings.OTEL_COLLECTOR_TAILNET_ENDPOINT`), not be hardcoded in the template.
- **New, corrected in this revision**: a manually-authored `Service` resource (not the Helm chart's own `service.enabled` toggle — see FD1's note on why that defaults to `internalTrafficPolicy: Local` in DaemonSet mode) selecting the OTel Collector DaemonSet's pods, with default (cluster-wide) traffic policy so it round-robins across whichever collector replica is reachable — edge devices don't need pod affinity to a specific node the way `hostIP`-based in-cluster services do, they just need one stable endpoint. This Service is then what gets exposed to the tailnet.
- **Tailnet exposure mechanism — narrowed in this revision**: the original draft listed three options (Ingress, Service annotation, Connector/ProxyGroup CRDs) without checking which actually applies. Verified against Tailscale's own operator docs: `ingressClassName: tailscale` is **Tailscale Serve** — built for HTTP(S) web apps/APIs with TLS termination, the wrong shape for raw OTLP gRPC (h2c, no TLS/ALPN needed). `Connector` is a subnet-router/exit-node CRD, not a single-Service-exposure mechanism. The correct mechanism is the `tailscale.com/expose: "true"` annotation (or `loadBalancerClass: tailscale`) on the new Service above — this operates via `iptables`/`nftables` DNAT below the application layer, protocol-agnostic, confirmed to work for arbitrary TCP/gRPC traffic.
- **Authentication — resolved via a dedicated Plan-phase investigation, defense in depth, both layers real and concrete, not a single either/or pick**: 1. **Application layer**: a *second*, dedicated OTLP receiver instance (`otlp/tailnet`, new ports — not the existing `:4317`/`:4318` every in-cluster service already uses) protected by the `bearertokenauth` extension — confirmed compiled into this repo's exact pinned collector image (`otelcol-contrib` `v0.105.0`) by reading that release's own component manifest. A shared token, stored the same way every other secret in this repo is (new `secrets/otel-collector.enc.env`, SOPS+age, decrypted into `monitoring/overlays/local/.secrets/` by the existing `secrets` role's pattern), checked against the standard `Authorization: Bearer <token>` header — which is exactly what the OTel SDK's own `OTEL_EXPORTER_OTLP_HEADERS` env var is designed to set, so the edge-device side (Feature Design 2b's Ansible template) needs no new mechanism, just one more env var line next to `OTEL_EXPORTER_OTLP_ENDPOINT`. A *second* receiver rather than adding `auth:` to the existing one, specifically so every already-wired in-cluster service is completely unaffected — bolting auth onto the shared receiver would 401 all of them unless every Deployment also threaded the token through, real scope creep this design avoids entirely. 2. **Network layer**: a Tailscale ACL tag (`tailscale.com/tags: "tag:otel-collector"`, same `service.annotations` block as the `expose` annotation) restricting which tagged devices can reach the exposed Service at all — this repo already has live precedent for tag-based device identity (`bluespot` advertises `tag:remote-controllers` on every field device's provisioning, confirmed by direct read of `bluespot/src/apps/registration/services/tailscale.py`), just not yet an ACL grant using it for anything. The actual grant is a manual step in the Tailscale admin console's policy file (this repo has no checked-in ACL policy at all today, confirmed by search — same as how the operator's own OAuth client credentials are already provisioned outside this repo's git history), not something a Kustomize/Ansible change can apply — state this plainly as an external, human setup step wherever this is documented, not something CI/`kubectl kustomize` can verify. See Feature Design 2c for the concrete file-level design.

## Feature Design 2c — OTLP-tailnet-receiver authentication (new, resolves FD2b's auth gap)

**Files**: `monitoring/overlays/local/values/otel.yml` (new `otlp/tailnet` receiver + `bearertokenauth` extension + `extraEnvs` + new `ports:` entries, alongside the existing `otlp` receiver, unchanged), `monitoring/overlays/local/kustomization.yml` (new `secretGenerator` block — this file has none today, confirmed by direct read), `ansible/roles/secrets/tasks/main.yml` (new decrypt step, mirroring the existing `tailscale.enc.env`/`orchestrator.enc.env` entries exactly), new `secrets/otel-collector.enc.env`; `orchestrator/src/settings/common.py` (new `OTEL_COLLECTOR_BEARER_TOKEN` setting, same pattern as the new `OTEL_COLLECTOR_TAILNET_ENDPOINT` setting from Feature Design 2b), `orchestrator/src/apps/components/ansible/templates/controllers.py` (new `OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer {{ otel_collector_bearer_token }}"` line in `EDGE_CONTROLLER_TMPL`'s `environment:` block).

Config shape (verified against `config/configauth`'s README at the matching opentelemetry-collector-contrib core tag, and `bearertokenauthextension`'s own source — server-side `Authenticate()` checks an `Authorization`/gRPC-metadata value against `"Bearer <token>"`, exactly the standard shape `OTEL_EXPORTER_OTLP_HEADERS` sets):

```yaml
extensions:
  bearertokenauth:
    token: ${env:OTEL_COLLECTOR_BEARER_TOKEN}

receivers:
  otlp/tailnet:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4319
        auth:
          authenticator: bearertokenauth
      http:
        endpoint: 0.0.0.0:4320
        auth:
          authenticator: bearertokenauth

service:
  extensions: [bearertokenauth]
  pipelines:
    traces:
      receivers: [otlp, otlp/tailnet]   # both feed the same existing pipeline
```

New `ports:` entries for `4319`/`4320` need `containerPort` **but deliberately no `hostPort`** — this path is meant to go only through Feature Design 2b's Service, not the existing hostPort mechanism every in-cluster service already uses.

## Feature Design 3a — `mediamtx`: enable native Prometheus metrics

**Files**: `plevion-k8s/services/mediamtx/base/mediamtx.yml`, `deployment.yml` (port 9998), `monitoring/overlays/local/values/prometheus.yml` (new scrape job — **not** the OTel Collector's own `prometheus` receiver, corrected below).

Add `metrics: yes` / `metricsAddress: :9998` to `mediamtx.yml` — verified directly against upstream MediaMTX v1.19.1's own `mediamtx.yml` reference (the exact version this repo's `Dockerfile` pins): real, documented, Prometheus-compatible, no custom code needed.

**Scrape mechanism — corrected in this revision**: the original draft's "likely right call" (route mediamtx's metrics through the OTel Collector's own `prometheus` receiver, for consistency with everything else going through one collector) is wrong for this specific case — the collector runs as a **DaemonSet**, so every node's collector instance would independently scrape mediamtx's single pod, producing duplicate metric series on any multi-node cluster (invisible on today's single-node microk8s, a real bug the moment `dev` becomes a real multi-node target). Use a direct Prometheus scrape job instead (`prometheus.yml`'s existing `extraScrapeConfigs` pattern already has one job for `otel-collector` pods specifically — add a second, equivalently scoped, job targeting mediamtx's pod label on port 9998).

**Defensive config addition, new in this revision**: `mediamtx.yml` doesn't currently set `authHTTPExclude`, so it inherits upstream's default (`action: api`, `action: metrics`, `action: pprof` all excluded from the auth-hook path) — which is why enabling `metrics: yes` is safe today without also routing metrics scrapes through `authmtx`'s `/auth` handler (which has no route for that and would break). This is currently an *implicit* dependency on an upstream default, not a decision this repo made explicitly. Add `authHTTPExclude` explicitly, listing `api`/`metrics`/`pprof`, so it's self-documenting and doesn't silently break if anyone later sets `authHTTPExclude` for an unrelated reason and overwrites the inherited default.

## Feature Design 3b — `mediamtx`: shared-trace correlation via `$MTX_SOURCE_ID`

**Corrected twice now** — see Confirmed Scope Decision 4 above for the full history (root-span-plus-attributes, then upgraded to genuine shared-trace correlation via a second Plan-phase investigation). Concrete mechanism, grounded in MediaMTX's own source at the pinned tag (`v1.19.1`), not assumed:

- **`services/mediamtx/base/mediamtx.yml`**: add `&source_id=$MTX_SOURCE_ID` to both existing `runOnReady`/`runOnNotReady` `wget` URLs (currently pass `path`/`query`/ `source_type` only). `$MTX_SOURCE_ID` is the same UUID MediaMTX already sends on every `/auth` call's JSON body (`id` field) — confirmed identical across a connection's `/auth` → `runOnReady` → `runOnNotReady` lifecycle by tracing `c.uuid` through `internal/servers/rtmp/conn.go` (generated once, at connection-object creation) and `internal/hooks/on_ready.go` (the same closure/env reused for both hook firings). One-line config change, no image/binary change.
- **`authmtx`**: add `github.com/google/uuid` (tiny, no transitive deps). In each handler (`auth.go` already decodes the ID from the JSON body; `ready.go`/`stop.go` need to read the new `source_id` query param), parse the UUID and construct a deterministic `trace.SpanContext` from its 16 bytes (`uuid.UUID` and `trace.TraceID` are both `[16]byte` — direct reinterpretation), inject it as a remote parent context, then start each handler's real span from that context. This is handler-level work, not something the generic `otelhttp.NewHandler(mux, ...)` middleware from Feature Design 4 can do on its own (that middleware runs before the request body/query params are parsed) — additive to it, not a replacement.
- **Result**: `auth`/`ready`/`notready` for one MediaMTX connection become three spans sharing one real, queryable Tempo trace ID — open by trace ID, see the full lifecycle. The synthetic parent span ID won't correspond to a real recorded span, so Tempo will likely render these as top-level spans within that one trace rather than a clean parent→child tree — verify the exact rendering empirically during Build, don't assume a particular visual shape.
- **Still not achievable, unchanged**: no nesting under `sink`/`orchestrator`'s own trace for the same stream (no inbound trace context exists to link them at the RTSP/WebRTC/SRT layer) — cross-referencing those still happens by looking up the shared object ID in Tempo.

## Feature Design 4 — `authmtx`: OTel Go SDK instrumentation

**Files**: `go.mod` (new deps), `cmd/authmtx/main.go` (SDK init/shutdown), `internal/server/server.go` (wrap the `mux` with `otelhttp.NewHandler`), `internal/upstream/client.go` (wrap the outbound HTTP client's `Transport` with `otelhttp.NewTransport`), `services/mediamtx/base/authmtx.yml` (stop hardcoding `service.version`/`service.environment`, read from env instead — see below).

**Dependency-cost correction, new in this revision**: `authmtx/go.mod` today has exactly one dependency (`gopkg.in/yaml.v3`). The original draft called adding OTel instrumentation "small, well-scoped... not a broad dependency expansion" — this understates it. The gRPC-based exporter (`otlptracegrpc`) pulls in `google.golang.org/ grpc`, `google.golang.org/protobuf`, `genproto`, and several more transitive modules — dozens of new entries in `go.sum`. **Use `otlptracehttp` instead** — the collector already has its OTLP HTTP receiver enabled (`:4318`, confirmed in `otel.yml`) alongside gRPC, and the HTTP exporter avoids pulling in the entire gRPC/protobuf tree. State the real module-count tradeoff honestly in Plan rather than calling it "small."

`internal/server/server.go`'s existing structure (`mux` → `recoveryMiddleware` → `http.Server`) makes this a clean wrap: `otelhttp.NewHandler(wrapped, "authmtx")` around the existing middleware chain. `internal/upstream/client.go`'s outbound call to Kong gets `otelhttp.NewTransport` on its `http.Client.Transport`.

**Resource attributes, corrected in this revision — this was Open Question 6 in the original draft, now answered by direct verification, not left open**: `services/mediamtx/base/authmtx.yml` already has `service: {name: "authmtx", version: "latest", environment: "local"}` — `name` is fine, but `version`/`environment` are **hardcoded literals**, not read from Phase 1's `SERVICE_VERSION`/`PLEVION_VERSION` env vars (which **are** already injected into this container via `configMapKeyRef: {name: plevion-platform-version, key: authmtx}` — confirmed directly, no gap there). The actual work is in `internal/config`: read `SERVICE_VERSION`/`PLEVION_VERSION` from env and use them instead of the config file's hardcoded strings.

## Feature Design 5 — `kong`: enable tracing — corrected target, this revision

**The original draft's Files line (`kong.yml`) was wrong and would have shipped a no-op.** Verified: `plevion-k8s` deploys Kong via `env.database: "off"` + `ingressController.enabled: true` (`services/kong/base/values.yml`) — config comes from Kubernetes CRs (`KongClusterPlugin`, following the exact precedent already in this repo at `services/overlays/local/auth-plugin.yml`), not a declarative file. The `kong` repo's own `kong/kong.yml` is dead, stale docker-compose-era config (`url: "http://mediamtx:8889"`, `auth_url: "http://auth-service:8081"` — neither resolves in this cluster) that the Kong image doesn't even `COPY` — it's not consumed anywhere in `plevion-k8s`.

**Files (corrected)**: new `services/overlays/local/otel-plugin.yml` (a `KongClusterPlugin`, `plugin: opentelemetry`, mirroring `auth-plugin.yml`'s shape — `config.endpoint` pointed at the collector's OTLP HTTP receiver, `:4318/v1/traces` — verify in Plan whether this Kong version's plugin schema wants HTTP or gRPC, don't assume); `services/kong/base/values.yml`'s `env:` block (new: `KONG_TRACING_INSTRUMENTATIONS` and `KONG_TRACING_SAMPLING_RATE` — **required**, confirmed against Kong's own plugin docs: *"Kong Gateway creates a top-level span for each request... when `tracing_instrumentations` is enabled"* — the plugin alone, without these, produces no spans at all). Also verify `config.propagation` vs. the deprecated `config.header_type` against Kong 3.5's actual accepted schema (`header_type` is deprecated as of a version at or before this repo's pin — confirm exactly).

**Rationale, corrected in this revision**: the original draft claimed "without this, every downstream service's traces start fresh at Kong regardless of how well instrumented they are individually" — **false**. Kong forwards inbound request headers to upstreams by default; a `traceparent` header from a frontend's instrumented request already reaches Django's own instrumentation whether or not this plugin is enabled. What the plugin actually adds is a **Kong-level span** in the trace (visibility into Kong's own request-handling time, not the thing that makes propagation possible at all). Enabling it is still the right call — Kong is the single entry point for all `/public/v1/*` traffic and a Kong-side span is genuinely useful — just state the real reason, not an invented one.

## Feature Design 6 — Resource-attribute standardization (OpenTelemetry semantic conventions)

**Files**: every service touched above, `frontend-landing`/`frontend-admin`'s `instrumentation.node.ts`.

- `service.name` — already correct everywhere it's set (`plevion-landing`, `plevion-admin`; the rest come from closing Feature Design 2a/2b's gaps).
- `service.version` — tie to the existing `SERVICE_VERSION` env var (Phase 1's `plevion-platform-version` ConfigMap injection) rather than inventing a parallel mechanism — already the right call, unchanged from the original draft.
- `deployment.environment.name` — **new caveat, this revision**: `frontend-landing`/ `frontend-admin` pin `@opentelemetry/semantic-conventions@^1.25.1`. `ATTR_DEPLOYMENT_ENVIRONMENT_NAME` is an **incubating** export (available only via the `/incubating` subpath) added after that pin — not importable from the stable entrypoint as currently pinned. Plan needs to decide: bump the semantic-conventions package, import from the incubating subpath, or use the raw attribute-key string directly rather than the constant. Don't assume the two-key `Resource({...})` edit the original draft implied is a trivial one-liner.

**Log noise vs. trace correlation — new section, this revision, resolving Spec Revision finding 14 (the original draft dropped this entirely)**: `monitoring/overlays/ local/values/grafana.yml`'s Loki datasource does log-to-trace linking via `derivedFields: matcherRegex: "trace_id=(\\w+)"` — this depends on `opentelemetry-instrumentation-logging` injecting `trace_id=` into every log line, which is *exactly* the kind of "OTel-added prefix/suffix noise" `ROADMAP.md`'s log- readability bullet wants stripped. These two ROADMAP asks are in direct tension and must be resolved together, not separately:
- **Resolution**: don't disable the logging instrumentation package (that would break Grafana's derived-field link entirely). Instead, address log noise in the OTel Collector's own log-processing pipeline (a `transform` processor trimming cosmetic/redundant fields from the rendered log line) while leaving the `trace_id=`-bearing structure Grafana depends on intact. This needs to be verified against the collector's actual current log-processing config in Plan — don't assume a specific processor without checking what's already there.
- **Logs-via-OTLP, explicitly out of scope, new in this revision**: the collector's `logs` pipeline currently has only a `filelog` receiver — no `otlp` receiver (the config file's own comment notes this: "If you wanted to receive OTLP logs from the app, add 'otlp' here"). Any log records a service's OTel SDK exports directly over OTLP are silently dropped today; all logs actually arrive via node-file scraping instead, which carries `k8s.*` resource attributes, not `service.*` ones. This spec does **not** propose adding an `otlp` logs receiver — `ROADMAP.md`'s ask is about `kubectl logs`/Loki readability, which the existing `filelog` path already serves. State this as a deliberate boundary, not an oversight, if it comes up in Plan.

## Environments in scope

`local` primarily. `dev` parity is explicitly **not** attempted by this spec beyond the guard in Feature Design 1 (fail clearly, don't silently break, if `--with-monitoring --env=dev` is invoked before `monitoring/overlays/dev/` exists) — full `dev` telemetry parity remains a separate, not-yet-scoped follow-up per `ROADMAP.md`'s own "dev is partial" item.

## Assumptions

- `opentelemetry-bootstrap` has run (or needs to be added) per-repo for the already-wired services — verify, don't assume, per Feature Design 2a.
- The OTel Collector's existing `otlp` receiver (`:4317` grpc, `:4318` http) needs no reconfiguration to accept traffic from newly-instrumented services.
- OTLP exporters in the Python/Go SDKs fail non-fatally (background retry, not startup/request-blocking) when the collector is unreachable — assumed based on standard SDK behavior, not yet empirically confirmed against this repo's actual configuration; confirm in Plan/Build (see Feature Design 1's default-off note).

## Testing Strategy

- Offline-verifiable: `ansible-playbook site.yml --syntax-check` after Feature Design 1's new role; `yamllint`/`kubectl kustomize --enable-helm` after every manifest change.
- Live-verifiable against the existing local microk8s cluster, no new credentials needed: every piece here, once `--with-monitoring` exists.
- `authmtx`/Kong changes: `go test -race ./...` / existing manual `docker compose up` verification steps, plus a live check that a real span appears in Tempo for an actual MediaMTX auth/ready/stop cycle and a real Kong-routed request.

## Boundaries

- **Always**: reuse the existing `SERVICE_VERSION`/Phase-1-ConfigMap mechanism for `service.version` rather than inventing a parallel one; follow the `--with-frontend` precedent's exact shape for `--with-monitoring`; verify Kong's plugin schema and MediaMTX's config keys against their real docs/source before writing config, not from memory or from what a prior draft of this spec assumed.
- **Ask first**: whether re-running `bootstrap.sh` without `--with-monitoring` after a prior `--with-monitoring` run should tear down the monitoring stack or leave it. (The OTLP-receiver-on-the-tailnet authentication gap that was previously listed here is now resolved — see Feature Design 2c — but the *Tailscale ACL grant itself* is a manual, external admin-console step no file in this repo can apply or verify; treat that step, specifically, with the same "ask first"/confirm-before-relying-on-it care this repo's `CLAUDE.md` already applies to networking/exposure decisions.)
- **Never**: treat "the OTel packages are listed in `pyproject.toml`" as equivalent to "this service emits telemetry" — get this from reading the actual k8s Deployment command/env, not the repo's own dependency file or its `CLAUDE.md`'s claims (both were wrong, in different directions, for services audited during this spec's Revision pass); bump `monitoring/overlays/local/values/otel.yml`'s pinned collector image tag as a drive-by while touching this file for other reasons — the pin is load-bearing for the (deprecated but still-required-at-this-version) `loki` log exporter, see Feature Design 1.

## Open Questions

1. Feature Design 1's transition-away behavior (teardown vs. leave monitoring data).
2. ~~Feature Design 2b's OTLP-receiver-on-tailnet authentication design~~ — **resolved**, see Feature Design 2c. The Tailscale ACL grant's manual admin-console step remains an execution-time dependency, not a design question.
3. Feature Design 6's log-noise-reduction mechanism in the OTel Collector pipeline — needs to be checked against the pipeline's actual current config, not designed in the abstract.
4. ~~Whether a live Tempo-trace-arrival check should be built as a repeatable preflight/postflight ansible detector or done as a one-time manual check~~ — **resolved**: build the repeatable detector, matching Phase 1's self-healing pattern, per explicit user direction ("a workload needs the same checks as main services workload"). See the Plan for the concrete preflight/postflight task design (Tempo's real search API, port 3100 per this repo's actual pinned chart — verified, not the commonly-cited 3200 default).
5. Full `dev` environment telemetry parity — explicitly out of scope for this phase (see "Environments in scope"), tracked as a separate follow-up.

## Revision Changelog

Independent Opus 5 Spec Revision pass found:
1. **Factual error, blocking**: the original "empirical audit" claimed `auth`/ `orchestrator`/`sink` had zero runtime OTel wiring — false; all three are wired via `opentelemetry-instrument` command overrides in their k8s Deployments, a location the original audit didn't check. Corrected: rewrote the audit into a per-workload gap table, most of which already works once Feature Design 1 exists.
2. **Root-cause reframing**: "monitoring is never deployed" was correctly identified but treated as a standalone convenience gap rather than the actual reason nothing works today. Corrected: stated explicitly as the blocking prerequisite for every other Feature Design.
3. **FD5 (Kong) factual error, blocking**: original target file (`kong.yml`) is dead, unused config from a different (docker-compose) deployment context — real Kong config is via `KongClusterPlugin` CRDs. Corrected target file and added the required `KONG_TRACING_INSTRUMENTATIONS`/`KONG_TRACING_SAMPLING_RATE` env vars the plugin needs to actually produce spans (omitted entirely from the original draft). Also corrected the stated rationale (Kong already forwards headers by default; the plugin adds a Kong-level span, doesn't enable propagation).
4. **FD3b/CSD4 design error**: "spans nest inside the existing distributed trace" is not achievable — MediaMTX's hooks are subprocess execs with no trace-context propagation path. Corrected to attribute-based correlation (root spans + shared ID lookup), and the internal inconsistency between CSD4's "confirmed answer" framing and FD3b's own hedge (Open Question 3 in the original draft) is resolved by this correction rather than left open.
5. **FD2b feasibility gaps**: no OTel Collector Service exists for the tailnet-exposure design to attach to (in-cluster reachability is via `hostIP`/`hostPort`, not a Service); the "trivial" in-cluster edge-controller case actually has no wrapper and no OTel deps at all; the Tailscale exposure mechanism was left as three undifferentiated options when only one (Service annotation) is actually correct for raw OTLP gRPC; no authentication/exposure-risk discussion for a tailnet-reachable telemetry write endpoint. All corrected/added.
6. **FD1 missing edge case**: `monitoring/overlays/dev` doesn't exist, so `--with-monitoring --env=dev` (a reachable flag combination) would hard-fail — added as a guard requirement, not left as a soft Open Question.
7. **FD1 missing consequence**: off-by-default monitoring means every already-wired service's OTLP exporter connection-refuse-loops by default — noted explicitly, including its plausible connection to the log-verbosity complaint this same ROADMAP item is trying to fix.
8. **FD3a feasibility error**: routing mediamtx's Prometheus metrics through the OTel Collector's own `prometheus` receiver would duplicate-scrape under the collector's DaemonSet deployment mode on any multi-node cluster. Corrected to a direct Prometheus scrape job. Also added the `authHTTPExclude` defensive-config note (currently an implicit, unstated dependency on an upstream default).
9. **FD4 honesty correction**: "small, well-scoped, not a broad dependency expansion" understated adding a gRPC-based OTLP exporter to a one-dependency Go module by a large margin. Corrected to recommend the HTTP exporter instead (avoids the gRPC/protobuf transitive tree) and to state the tradeoff honestly either way.
10. **Open Question 6 (original) answered, not left open**: `SERVICE_VERSION`/ `PLEVION_VERSION` already reach `authmtx`'s container via the existing Phase 1 ConfigMap mechanism — confirmed directly. The actual gap is narrower and different than the question implied: `services/mediamtx/base/authmtx.yml` hardcodes `service.version`/`service.environment` as literals instead of reading the already-present env vars.
11. **Scope gap**: the original draft addressed only 2 of `ROADMAP.md`'s 4 telemetry bullets, without stating that narrowing in the Objective. Found (finding 14) that the other two are directly coupled to what's already in scope (log-noise stripping would break the Loki-to-Tempo correlation this spec's own Feature Design 6 depends on) — folded in a scoped resolution rather than leaving the tension unaddressed.
12. **New Boundary items**: don't bump `otel.yml`'s pinned collector image tag as a drive-by (load-bearing for the deprecated-but-required `loki` exporter at this version); don't trust either a repo's `pyproject.toml`/`go.mod` dependency list or its own `CLAUDE.md` claims as evidence of actual runtime telemetry behavior again.

### Second pass — Plan-phase investigation (user follow-up, not a formal Spec Revision round)

Triggered by three user questions raised while reviewing the Plan: (1) wanting a real authentication mechanism for the tailnet-exposed OTLP receiver rather than accepting the exposure as a scoped risk, (2) wanting the `monitoring` workload to get the same preflight/postflight self-healing checks every other workload already has, and (3) asking whether `mediamtx`'s own custom image could inject real tracing rather than settling for attribute-based correlation. A dedicated investigation (general-purpose agent, both local source reading and external doc/source fetches) resolved all three:

13. **New Feature Design 2c**: a dedicated `otlp/tailnet` receiver protected by the collector's `bearertokenauth` extension (confirmed compiled into the exact pinned `otelcol-contrib` v0.105.0 image) plus a Tailscale ACL tag — defense in depth, doesn't touch the existing in-cluster OTLP path at all. Resolves Feature Design 2b's previously-open authentication gap.
14. **Feature Design 3b upgraded a second time**: the MediaMTX-trace-correlation design (root-span-plus-attributes from the first revision) turned out to be more conservative than necessary. Tracing MediaMTX's own source at the pinned tag found a stable per-connection UUID (`$MTX_SOURCE_ID`) already flows to `authmtx` today (unused) and, with a one-line config addition, can also reach the `runOnReady`/ `runOnNotReady` hooks — enabling a deterministic shared trace ID across a connection's full auth/ready/notready lifecycle. No image or binary change needed, contrary to the premise of the question that triggered this investigation.
15. **Open Question 4 (verification mechanism) resolved by explicit user direction**, not left as a spec-level tradeoff: build the repeatable preflight/postflight detector. The investigation additionally corrected the Tempo query port — this repo's vendored chart pins `3100`, not the commonly-cited `3200` default — and confirmed the Ansible control node has no direct route to in-cluster `*.svc.cluster.local` DNS on this topology, so the live check needs to run as an in-cluster throwaway pod, not a direct `ansible.builtin.uri` call from the control node the way the spec's original phrasing implied.