# PLAN — Telemetry Cleanup **Based on**: `plevion-docs/SPEC_TELEMETRY_CLEANUP.md` (revised once, independent Opus 5 Spec Revision pass incorporated). **Written against**: same repo tips as the spec, plus direct verification of every file this plan touches (see "Plan-time corrections" below for two places where reading the actual current state refined the spec's design). ## Goal Land the spec's Feature Designs 1-6 across `plevion-k8s`, `orchestrator`, `edge- controller`, `authmtx`, `frontend-landing`, `frontend-admin` — monitoring actually deploys (opt-in), the small enumerable set of Python instrumentation gaps close, `mediamtx` gets native metrics, `authmtx` gets real tracing (and MediaMTX gets attribute-based correlation through it), Kong gets a real tracing config (not the dead file the original spec draft targeted), and resource-attribute tagging is standardized without breaking Grafana's existing log-to-trace correlation. ## Plan-time corrections to the spec (verified against real chart/source, not assumed) 1. **FD1/FD2b's "new Service" — simpler than the spec proposed.** The spec called for a manually-authored Service (to avoid the Helm chart's own `service.enabled` toggle defaulting to `internalTrafficPolicy: Local` in DaemonSet mode). Verified directly against the vendored chart source (`monitoring/overlays/local/charts/opentelemetry-collector-0.140.1/.../_helpers.tpl`): the chart supports `service.internalTrafficPolicy: Cluster` as an explicit override, which sidesteps the DaemonSet default entirely. Use the chart's own `service:` values block (`enabled: true`, `internalTrafficPolicy: Cluster`) instead of a separately-authored resource — same outcome, one less resource to maintain, stays inside this repo's established values-file-driven pattern. 2. **FD2b's tailnet exposure — the same values block, not a separate step.** The chart's `service.yaml` template renders `service.annotations` verbatim onto the Service it creates. `tailscale.com/expose: "true"` goes in that same `service.annotations` map in `otel.yml` — no second resource, no separate task. ## Risks | Risk | Mitigation | |---|---| | `otel.yml`'s pinned collector image (`0.105.0`) is load-bearing for the deprecated `loki` exporter — Tasks 2, 2b, and 6 all touch this file, each risks a drive-by version bump | Explicit Boundary (below), called out independently in each of the three tasks | | Enabling `service: {enabled: true}` on a DaemonSet-mode chart is new, unexercised in this repo | Task 2's acceptance criteria requires confirming the Service actually round-robins across collector replicas (not just "exists") before relying on it for the tailnet path | | `--with-monitoring --env=dev` is a reachable flag combination with no `monitoring/overlays/dev/` to build | New role must assert and fail clearly (Task 1), not let kustomize produce a raw error | | Bolting auth onto the *existing* `:4317`/`:4318` OTLP receiver would 401 every already-wired in-cluster service | Task 2b uses a second, dedicated `otlp/tailnet` receiver on new ports instead — the shared in-cluster receiver is untouched, confirmed as an explicit acceptance check | | The Tailscale ACL grant (Task 2b) is a manual, external admin-console step no file in this repo can apply or verify — a missing grant fails silently at the network layer | Explicit, separately-called-out Checkpoint in Task 2b and re-confirmed in Checkpoint 3 — don't treat edge-device telemetry as working from green CI/kustomize checks alone | | `edge-controller`'s `deployment.yml` wrapper change (Task 3) depends on `edge-controller`'s own repo (Task 8) shipping the new OTel dependencies first, or the wrapped command fails with "command not found" | Sequence: Task 8 (edge-controller repo) merges and a new image builds before Task 3's edge-controller manifest change is verified live — Task 3 can still be *authored* in parallel | | `authmtx`'s tracing (Task 10) and `mediamtx`'s `source_id` plumbing (Task 4) are independently mergeable but both needed for the shared-trace-ID design to actually show anything in Tempo | No hard blocking dependency — but note in Checkpoint 3 that both need to be live before that acceptance criterion (three spans, one trace ID) can be checked | | Task 10's deterministic-`SpanContext` construction from a UUID is a real, if small, cryptographic-adjacent design choice (reusing connection UUID bytes as trace ID bytes) — worth a second look during Code Revision, not just Build | Flagged explicitly here so Code Revision doesn't skip past it as boilerplate OTel wiring | | Log-noise reduction (Task 6) risks breaking Grafana's `derivedFields` trace-log link if done naively (stripping `trace_id=` along with genuine noise) | Task 6 is explicitly an investigation-first task — read the current `transform` processor's actual behavior before proposing a specific fix, not designed in the abstract | | Task 6.5's postflight Tempo check assumes the collector/Tempo images have no `curl`/`wget` on `PATH` (plausible for `otelcol-contrib`'s minimal image, unverified for Tempo's own) | Uses a throwaway `kubectl run --rm --image=curlimages/curl` pod instead of `kubectl exec` into an existing service pod, sidestepping the question entirely | ## Task Breakdown Tasks 1-6.5 (1, 2, 2b, 3, 4, 5, 6, 6.5) are `plevion-k8s` — per this repo's established Infra Build convention (`feedback_infra_build_stack_commits_one_pr`), stack all as commits on **one branch, one PR**, iterating live against the local cluster rather than one-PR-per-task. Tasks 8-12 are each their own repo, each its own PR (matching the k8s-deployment-hardening Phase 2 precedent for cross-repo mechanical/infra work). --- ### Task 1 — `plevion-k8s`: `--with-monitoring` flag + `monitoring` ansible role **Files:** `ansible/bootstrap.sh`, `ansible/site.yml`, new `ansible/roles/monitoring/tasks/main.yml`. - [ ] `bootstrap.sh`: new `--with-monitoring` flag, `WITH_MONITORING="false"` default, threaded through exactly like `--with-frontend`/`WITH_FRONTEND` (same case-statement shape, same `-e "with_monitoring=${WITH_MONITORING}"` extra-var). - [ ] `site.yml`: new `with_monitoring: false` default var; new `monitoring` role in `roles:`, `when: with_monitoring | bool`, placed after `infrastructure` and before `migrations` (no dependency either direction found — `monitoring` targets its own namespace, nothing else reads from it at apply time). - [ ] New role: guard first — `ansible.builtin.stat` (or equivalent) on `monitoring/overlays/{{ env }}`; if missing, `ansible.builtin.fail` with a clear message naming the env and that only `local` currently has a monitoring overlay (closes the `--with-monitoring --env=dev` gap the spec flagged as unguarded). - [ ] Build + apply `monitoring/overlays/{{ env }}` (same `kubectl kustomize ... --enable-helm` + `kubernetes.core.k8s` pattern every other role uses). - [ ] Wait for the OTel Collector DaemonSet to have ≥1 `Ready` pod **and** verify the wait actually checks the path services use — `hostIP:4317` reachability (a real TCP check, e.g. via `ansible.builtin.wait_for` against a node IP), not just "pod phase is Running," which was Spec Revision's specific critique of a naively-scoped readiness check. **Acceptance:** - [ ] `ansible-playbook site.yml --syntax-check` passes. - [ ] `./ansible/bootstrap.sh --with-monitoring` on a cluster with nothing running yields a `Ready` OTel Collector DaemonSet pod on every node, confirmed via `microk8s kubectl get pods -n monitoring`. - [ ] `./ansible/bootstrap.sh --with-monitoring --env=dev` fails with the new explicit guard message, not a raw kustomize path error. - [ ] Plain `./ansible/bootstrap.sh` (no flag) does not attempt to build/apply `monitoring/` at all. --- ### Task 2 — `plevion-k8s`: OTel Collector Service + tailnet exposure **Files:** `monitoring/overlays/local/values/otel.yml`. - [ ] Add a `service:` block: `enabled: true`, `internalTrafficPolicy: Cluster` (see "Plan-time corrections" above — verified against the vendored chart's `_helpers.tpl`, this sidesteps the DaemonSet-mode `Local` default without a separately-authored resource). - [ ] Add `service.annotations`: `tailscale.com/expose: "true"` **and** `tailscale.com/tags: "tag:otel-collector"` (the ACL-matchable tag Task 2b's grant targets — added here, not in 2b, since it's the same `service.annotations` map) — the chart's `service.yaml` template renders `service.annotations` directly onto the Service it creates, so exposure is the same edit as creating the Service, not a second step. - [ ] **Do not** touch `image.tag` in this file while here (Boundary — see below). **Acceptance:** - [ ] `microk8s kubectl get svc -n monitoring` shows the new collector Service, `internalTrafficPolicy: Cluster`. - [ ] `microk8s kubectl get svc -n monitoring -o yaml` shows both the `tailscale.com/expose` and `tailscale.com/tags` annotations; confirm (via `tailscale status` or the operator's own logs) a corresponding tailnet node/proxy actually provisions, tagged correctly. - [ ] A synthetic OTLP export from *outside* the cluster's pod network (e.g. from the dev machine itself, once on the tailnet) reaches the collector through this Service — full authenticated success only verifiable once Task 2b lands too. --- ### Task 2b — `plevion-k8s`: OTLP-tailnet-receiver authentication **Files:** `monitoring/overlays/local/values/otel.yml` (new `otlp/tailnet` receiver + `bearertokenauth` extension + `extraEnvs` + new `ports:` entries — the existing `otlp` receiver on `:4317`/`:4318` is untouched), `monitoring/overlays/local/kustomization.yml` (new `secretGenerator` block — none exists today, confirmed by direct read), `ansible/ roles/secrets/tasks/main.yml` (new decrypt step, mirroring the existing `tailscale.enc.env`/`orchestrator.enc.env` entries), new `secrets/otel-collector.enc.env`. Resolves the auth gap Task 2 previously left open — see `SPEC_TELEMETRY_CLEANUP.md` Feature Design 2c for the full investigation and rationale (a dedicated general-purpose-agent research pass, confirmed `bearertokenauth` is compiled into this repo's exact pinned `otelcol-contrib` image `v0.105.0`). - [ ] New `secrets/otel-collector.enc.env` (SOPS+age, `OTEL_COLLECTOR_BEARER_TOKEN=` — `.sops.yaml`'s existing `path_regex` already matches this filename, no `.sops.yaml` change needed). - [ ] `ansible/roles/secrets/tasks/main.yml`: new decrypt+write step, same shape as the existing entries, writing to `monitoring/overlays/local/.secrets/otel-collector.env`. - [ ] `monitoring/overlays/local/kustomization.yml`: new `secretGenerator` block generating an `otel-collector-bearer-token` Secret from that file. - [ ] `otel.yml`: new `extensions.bearertokenauth.token: ${env:OTEL_COLLECTOR_BEARER_TOKEN}`; `extraEnvs` gets a new entry sourcing `OTEL_COLLECTOR_BEARER_TOKEN` from `secretKeyRef: {name: otel-collector-bearer-token, key: OTEL_COLLECTOR_BEARER_TOKEN}` (same `extraEnvs:` list already used for `K8S_NODE_NAME`); new `otlp/tailnet` receiver (grpc `0.0.0.0:4319`, http `0.0.0.0:4320`, both `auth.authenticator: bearertokenauth`), added to the existing `traces` pipeline's `receivers:` list alongside `otlp` (not a separate pipeline); new `ports:` entries for `4319`/`4320` with `containerPort` set but **no `hostPort`** (this path only goes through Task 2's Service, never the existing hostPort mechanism). - [ ] **Do not** touch `image.tag` in this file while here (same Boundary as Task 2/6). **Checkpoint — external, human, not verifiable by this repo's tooling**: the Tailscale ACL grant itself (`{"action": "accept", "src": ["tag:remote-controllers"], "dst": ["tag:otel-collector:4319", "tag:otel-collector:4320"]}`) has to be added to the Tailscale admin console's policy file by whoever manages that account — this repo has no checked-in ACL policy today. Confirm this has actually been done, with the user, before treating end-to-end edge-device telemetry as working — a missing ACL grant fails silently at the network layer, not with a clear error this repo's own tooling would surface. **Acceptance:** - [ ] `yamllint .` + `kubectl kustomize --enable-helm` pass. - [ ] Existing in-cluster OTLP traffic on `:4317`/`:4318` is provably unaffected (rerun Task 3's acceptance check). - [ ] A token-less export against `:4319`/`:4320` (through the tailnet Service) gets a 401/unauthenticated rejection. - [ ] A correctly-tokened export against the same ports succeeds. --- ### Task 3 — `plevion-k8s`: close the enumerable Python instrumentation gaps **Files:** `services/auth/base/deployment.yml`, `services/auth/base/gateways/events/ deployment.yml`, `services/orchestrator/base/deployment.yml`, `services/orchestrator/base/crons/{reap-connections,check-components-updates}-cron.yml`, `services/sink/base/gateways/broadcast/deployment.yml`, `services/sink/base/workers/sink-cqrs-worker.yml`, `services/edge-controller/base/deployment.yml`. Per-file, exactly what's missing (verified directly, not re-derived from the spec's own audit table — matches it): - [ ] `auth`'s two Deployments, `orchestrator`'s main Deployment: add `OTEL_SERVICE_NAME` env var (currently missing — `OTEL_EXPORTER_OTLP_ENDPOINT` is already present on all three). - [ ] `reap-connections-cron`/`check-components-updates-cron`: currently `command: ["python"]` — add `opentelemetry-instrument` prefix, plus `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_SERVICE_NAME` (neither Job currently has any OTel env at all — confirm and add both, don't assume only the wrapper is missing). - [ ] `sink`'s broadcast gateway: currently `command: ["gunicorn"]` — add `opentelemetry-instrument` prefix + confirm/add OTel env vars (same as the crons). - [ ] `sink-cqrs-worker`: currently `command: ["uv","run","python","manage.py", "cqrs_consume"]` — insert the wrapper **inside** the `uv run` invocation (`["uv","run","opentelemetry-instrument","python","manage.py","cqrs_consume"]`), not before `uv run` (which would try to instrument `uv` itself, not the Python process `uv` launches). `OTEL_SERVICE_NAME`/`OTEL_EXPORTER_OTLP_ENDPOINT` already present here — no env change needed, wrapper placement only. - [ ] `edge-controller`'s Deployment: currently `command: ["celery"]`, **no** OTel env vars at all. Add the wrapper + both env vars — **blocked on Task 8** (edge-controller repo) actually shipping the `opentelemetry-instrument` binary in its image first; can author this change now, verify live only after Task 8 merges and a new image builds. - [ ] Every Deployment/Job touched here (plus the ones already wired): add `OTEL_RESOURCE_ATTRIBUTES` with `deployment.environment.name=local` (or `dev`, per overlay) — the `service.version` piece is already covered by the existing `SERVICE_VERSION` `configMapKeyRef` pattern; this is purely the new `deployment.environment.name` attribute per Feature Design 6. **Acceptance:** - [ ] `yamllint .` + `kubectl kustomize environments/local/ --enable-helm` pass after every edit. - [ ] Once Task 1/2 are live: each service's `OTEL_SERVICE_NAME` shows up as a distinct `service.name` in Tempo's service list (query via Grafana's Tempo datasource or Tempo's own search API) within a few minutes of a real request hitting that service. --- ### Task 4 — `plevion-k8s`: `mediamtx` native metrics + shared-trace-ID plumbing **Files:** `services/mediamtx/base/mediamtx.yml`, `services/mediamtx/base/ deployment.yml`, `monitoring/overlays/local/values/prometheus.yml`. - [ ] `mediamtx.yml`: add `metrics: yes`, `metricsAddress: :9998`. Add `authHTTPExclude` explicitly (`api`, `metrics`, `pprof`) — currently relies on upstream's inherited default for this; make it explicit so it doesn't silently break if anyone later sets `authHTTPExclude` for an unrelated reason. - [ ] `mediamtx.yml`: add `&source_id=$MTX_SOURCE_ID` to both the `runOnReady` and `runOnNotReady` `wget` URLs (currently pass `path`/`query`/`source_type` only). This is the shared-trace-ID mechanism from `SPEC_TELEMETRY_CLEANUP.md` Feature Design 3b/2c — `$MTX_SOURCE_ID` is the same UUID MediaMTX already sends on every `/auth` call's JSON body (confirmed identical across a connection's full auth/ready/notready lifecycle by reading MediaMTX's own source at the pinned tag). One-line config change, no image/binary change — pairs with Task 10's `authmtx`-side handling. - [ ] `deployment.yml`: add `containerPort: 9998` to the `mediamtx` container's `ports:` list (currently missing). - [ ] `prometheus.yml`: new `extraScrapeConfigs` job mirroring the existing `otel-collector` job's shape exactly, but keyed on `__meta_kubernetes_pod_label_app` == `mediamtx` (not `app_kubernetes_io_instance` — `mediamtx`'s Deployment is manually-authored, not Helm-chart-managed, so it only carries a plain `app: mediamtx` label, confirmed by direct read) and port `9998`. **Not** via the OTel Collector's own `prometheus` receiver — confirmed in the spec this would duplicate-scrape under DaemonSet mode. **Acceptance:** - [ ] `yamllint .` + `kubectl kustomize --enable-helm` pass. - [ ] `curl http://:9998/metrics` returns Prometheus-format output after redeploy. - [ ] Prometheus's own targets page shows the new `mediamtx` job as `up`. - [ ] A real auth check against MediaMTX still succeeds (confirms `authHTTPExclude` didn't inadvertently change auth behavior for the metrics/api/pprof paths it's meant to exempt). --- ### Task 5 — `plevion-k8s`: Kong tracing **Files:** new `services/overlays/local/otel-plugin.yml`, `services/kong/base/values.yml`. - [ ] New `KongClusterPlugin` (`services/overlays/local/otel-plugin.yml`), mirroring `auth-plugin.yml`'s exact shape: `plugin: opentelemetry`, `config.endpoint` pointed at the collector's OTLP HTTP receiver (`http://.monitoring.svc.cluster.local:4318/v1/traces` — resolve the exact Service DNS name against what Task 2 actually creates). Verify Kong 3.5's actual accepted config keys before finalizing — confirm `config.propagation` (not the deprecated `config.header_type`) is correct at this pinned version. - [ ] Add this new resource to `services/overlays/local/kustomization.yml`'s `resources:` list (same way `auth-plugin.yml` is already listed there). - [ ] `services/kong/base/values.yml`'s `env:` block: add `KONG_TRACING_INSTRUMENTATIONS` and `KONG_TRACING_SAMPLING_RATE` — **required**, the plugin alone produces no spans without these (confirmed against Kong's own plugin docs). **Acceptance:** - [ ] `yamllint .` + `kubectl kustomize --enable-helm` pass. - [ ] A real request through Kong produces a Kong-attributed span in Tempo. --- ### Task 6 — `plevion-k8s`: log-noise investigation + `transform` processor **Files:** `monitoring/overlays/local/values/otel.yml` (the existing `transform` processor in the `logs` pipeline — already present, currently used only to copy resource attributes into flat `attributes` for Loki labeling, verified by direct read). - [ ] **Investigate first, don't design in the abstract**: capture a sample of what `kubectl logs`/Loki actually show for a real service today (once Task 1 is live) — confirm exactly which fields constitute the "noise" `ROADMAP.md` flags, and confirm the `trace_id=`-bearing structure Grafana's `derivedFields: matcherRegex: "trace_id=(\\w+)"` (in `values/grafana.yml`) depends on, so the fix doesn't remove it. - [ ] Extend the existing `transform` processor's `log_statements` to trim whatever the investigation identifies as genuine noise, leaving `trace_id=` (or wherever it actually lives in the log body/attributes once confirmed) untouched. - [ ] Do not touch `image.tag` in this file (same Boundary as Task 2 — this task and Task 2 both touch `otel.yml`, both need this reminder independently since they may be worked in either order). **Acceptance:** - [ ] Grafana's Loki → Tempo derived-field link still works for a real log line after the change (click-through from a log line to its trace in Tempo). - [ ] A human (not just a grep) confirms `kubectl logs`/Loki output is meaningfully more readable than before. --- ### Checkpoint 1 (after Task 2b) Confirm live, before proceeding to Tasks 3-6.5's verification steps: monitoring deploys cleanly via `--with-monitoring`, the collector Service exists with `Cluster` traffic policy, and Task 2b's tokened-vs-token-less acceptance checks both pass. The Tailscale ACL grant is a separate, external, manual step (see Task 2b's own Checkpoint) — don't treat full edge-device-to-cluster telemetry as verified until that's confirmed done too, even once everything in this repo's own tooling passes. --- ### Task 6.5 — `plevion-k8s`: monitoring preflight/postflight self-healing checks **Files:** `ansible/roles/preflight/tasks/main.yml`, `ansible/roles/postflight/tasks/main.yml`. Resolves the plan's previously-open verification-mechanism question by explicit user direction — "a workload needs the same checks as main services workload," i.e. build the repeatable detector, matching Phase 1's established preflight/postflight pattern, not a one-time manual check. Both new blocks gated on `kubectl get ns monitoring` succeeding, so the default (no `--with-monitoring`) path is completely unaffected. - [ ] **Preflight** (new block in the existing `Run drift detectors against the existing cluster` section, after the Kong-restart detector): list the OTel Collector DaemonSet's pods with restart counts via `argv`-form `kubectl get pods -n monitoring -l app.kubernetes.io/name=opentelemetry-collector -o jsonpath=...` (must be `argv:`, not `cmd:` — the jsonpath needs a literal space token `{" "}` that `cmd`'s shlex parsing mangles, the exact same gotcha Phase 1 already hit and fixed for its own restart-count detectors — copy that pattern, don't re-derive it). Threshold at `restartCount >= 4` (matching every other restart-count detector in this file), then `kubectl logs --previous` on any pod crossing it. Unlike the Tailscale-key/GHCR-PAT detectors, there's no known specific error string to pattern-match yet for this workload — detect-and-instruct here means surfacing the raw `--previous` log output as the failure message and pointing at `otel.yml`, not inventing a fake specific cause. - [ ] **Postflight** (new block, order-independent relative to the existing login smoke test): a live trace-arrival check. Since the Ansible control node has no route to in-cluster `*.svc.cluster.local` DNS on this microk8s topology (confirmed — every existing HTTP-shaped postflight check goes through `kubectl exec` or an external LoadBalancer IP, never direct `ansible.builtin.uri` to a cluster-internal address), implement via a throwaway `kubectl run --rm --image=curlimages/curl` pod (not exec'ing into an existing service pod — the collector/Tempo images are plausibly shell-less/distroless, unverified, don't assume `curl`/`wget` exist inside them) querying Tempo's real search API: `GET http://tempo.monitoring.svc.cluster.local:3100/api/search?tags=service.name%3Dauth&limit=1` — **port `3100`, verified against this repo's actual vendored Tempo chart values (`server.http_listen_port: 3100`), not the commonly-cited `3200` generic default**. Retry/poll (`until:`, a handful of attempts with a short delay) since trace export is async. Reuse the existing login smoke test as the stimulus that guarantees `auth-local` actually receives a real request worth tracing — this check runs after that stimulus, not as its own synthetic load generator. - [ ] Report result via `ansible.builtin.debug`, non-fatal (matching the login smoke test's own non-blocking pattern) — a missing trace after this phase's Tasks land is a real, actionable signal, but shouldn't hard-fail an otherwise-successful bring-up. **Acceptance:** - [ ] `ansible-playbook site.yml --syntax-check` passes. - [ ] A deliberately crash-looped otel-collector pod (e.g. temporarily bad `bearertokenauth` token) gets flagged by preflight with raw `--previous` log output. - [ ] A healthy `--with-monitoring` run's postflight prints a confirmed "trace found" message after the login smoke test fires. - [ ] Plain `./ansible/bootstrap.sh` (no `--with-monitoring`) shows zero new output from either new block — both skip cleanly via the namespace-existence guard. --- ### Task 8 — `edge-controller`: add OTel Python dependencies **Files:** `pyproject.toml`, `docker/dev.Dockerfile` (or wherever `opentelemetry-bootstrap` needs to run as a build step, matching whatever mechanism `auth`/`orchestrator`/`sink` actually use — verify against one of those repos' actual build process before assuming, since the spec flagged this as unconfirmed for the already-wired services too). - [ ] Add the same instrumentation package set `auth`/`orchestrator` carry: `opentelemetry-distro`, `opentelemetry-exporter-otlp`, `opentelemetry-instrumentation-django`, `-celery`, `-logging`, `-psycopg2`, `-redis` (edge-controller uses `django-redis`, `psycopg2-binary`, `celery`, `django-cqrs` — confirmed via `pyproject.toml` — so this instrumentation set is the right match, not a copy-paste guess). `-pika` only if CQRS transport is actually RabbitMQ-based here too (verify). - [ ] Add `opentelemetry-bootstrap` (or confirm it already runs some other way) as part of the image build, so `opentelemetry-instrument` (added to the k8s Deployment in Task 3, and to the real-device compose template in Task 9) actually has something registered to instrument. **Acceptance:** - [ ] `docker compose run --rm app_test` still passes. - [ ] A locally-built image has `opentelemetry-instrument` on `PATH` and `opentelemetry-bootstrap`'s registered instrumentors list includes Django/Celery/ psycopg2/redis. --- ### Task 9 — `orchestrator`: thread the OTLP endpoint + auth token through edge-device deployment **Files:** `src/settings/common.py` (new `OTEL_COLLECTOR_TAILNET_ENDPOINT` and `OTEL_COLLECTOR_BEARER_TOKEN` settings, env-var-backed, following the exact `SERVICE_NAME`/`API_URL` pattern already there), `src/apps/components/ansible/ templates/controllers.py` (`EDGE_CONTROLLER_TMPL`), `src/apps/components/ansible/ playbooks/components.py` (the `render_template(...)` call site, confirmed at the exact line that already sources `'api_url': settings.API_URL`). - [ ] New settings in `settings/common.py`, same style as `API_URL`/`SERVICE_NAME` — both env-var-backed, `OTEL_COLLECTOR_BEARER_TOKEN` sourced from wherever `orchestrator`'s own Deployment gets its secrets today (matching whatever mechanism already injects e.g. `CONTAINER_REGISTRY_TOKEN`, don't invent a new one). - [ ] `EDGE_CONTROLLER_TMPL`'s `environment:` block: add `OTEL_EXPORTER_OTLP_ENDPOINT: {{ otel_collector_endpoint }}`, `OTEL_SERVICE_NAME: edge_controller`, and `OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer {{ otel_collector_bearer_token }}"` (the standard OTel SDK env var for adding auth headers to OTLP exports — matches Task 2b's `bearertokenauth` extension on the collector side) alongside the existing `SERVICE_VERSION` line (exact same block, confirmed present at `controllers.py` line 11). - [ ] `EDGE_CONTROLLER_TMPL`'s `command:` — add `opentelemetry-instrument` prefix to the existing `celery -A settings worker ...` line (env vars alone do nothing without this — confirmed the same gap exists here as in Task 3's k8s-side fix, for the same reason). - [ ] `playbooks/components.py`'s render call: add `'otel_collector_endpoint': settings.OTEL_COLLECTOR_TAILNET_ENDPOINT` and `'otel_collector_bearer_token': settings.OTEL_COLLECTOR_BEARER_TOKEN` to the context dict, same place `api_url`/`service_name` are already sourced. **Acceptance:** - [ ] `docker compose run --rm app_test` passes. - [ ] A real (or staged) edge-device deployment's rendered `docker-compose.yml` includes the new env vars (including the auth header) and the wrapped command — verify by rendering the template directly (unit test or manual render), not just reading the diff. Confirm the bearer token never ends up logged/printed anywhere in this rendering path (it's a real secret now, not just a config value). - [ ] **Blocked on Task 2b/Checkpoint 1 and Task 8 for live verification**: an actual real-device trace landing in Tempo — not testable until the tailnet endpoint + auth exist and edge-controller's image has the OTel deps. Also blocked on the Tailscale ACL grant (Task 2b's external checkpoint) actually being applied. --- ### Task 10 — `authmtx`: OTel Go SDK instrumentation + shared-trace-ID correlation **Files:** `go.mod`, `cmd/authmtx/main.go`, `internal/server/server.go`, `internal/upstream/client.go`, `internal/handler/auth.go`, `internal/handler/ready.go`, `internal/handler/stop.go`. - [ ] `go.mod`: add `go.opentelemetry.io/otel`, `go.opentelemetry.io/otel/sdk`, `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` (**not** `otlptracegrpc`, confirmed in the spec — avoids pulling in the gRPC/protobuf transitive tree; the collector's `:4318` HTTP OTLP receiver is already enabled), `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp`, and `github.com/google/uuid` (tiny, no transitive deps — for the shared-trace-ID work below). - [ ] `main.go`: SDK init (resource with `service.name=authmtx`, `service.version`/`deployment.environment.name` — see Task 11 for where those values actually come from) + `TracerProvider` registration + graceful shutdown on the existing signal-handling path. - [ ] `server.go`: wrap the existing `wrapped := recoveryMiddleware(mux, logger)` result with `otelhttp.NewHandler(wrapped, "authmtx")` — confirmed this is a clean, minimal wrap given the current middleware-chain structure. This gives generic HTTP-server spans/metrics; it does **not** by itself produce the shared-trace-ID behavior below (it runs before the request body/query params are parsed) — additive to the per-handler work, not a replacement for it. - [ ] `upstream/client.go`: `NewHTTPClient`'s `&http.Client{Timeout: timeout}` — add `Transport: otelhttp.NewTransport(http.DefaultTransport)` (confirmed no `Transport` is currently set, so this is additive, not a conflicting override). - [ ] **Shared-trace-ID correlation (`SPEC_TELEMETRY_CLEANUP.md` Feature Design 3b)**: in each of `auth.go` (already decodes the source ID from the JSON body's `id` field — confirmed present, currently unused), `ready.go`/`stop.go` (read the new `source_id` query param Task 4 adds), parse it as a UUID and construct a deterministic `trace.SpanContext` from its 16 bytes (`uuid.UUID` and `trace.TraceID` are both `[16]byte` — direct reinterpretation, no hashing), inject it as a remote parent via `trace.ContextWithRemoteSpanContext`, then start each handler's real business-logic span from that context (`tracer.Start(ctx, "authmtx.auth")` / `"authmtx.ready"` / `"authmtx.notready"`). Result: the three calls for one MediaMTX connection's lifecycle share one real Tempo trace ID. On a parse failure (missing/ malformed source ID — e.g. a request that predates this change reaching a rolled-out service), fall back to a normal root span rather than erroring the request. **Acceptance:** - [ ] `golangci-lint run`, `go vet ./...`, `go test -race ./...` pass. - [ ] A manual `docker compose up` exercise (per this repo's own documented verification steps) against a live collector (once Task 1/2 land) produces a real span for a `/auth`, `/notify/ready`, and `/notify/stop` call each. - [ ] For three calls sharing the same `source_id`: confirm in Tempo that all three spans share one trace ID (open by trace ID, see all three) — not the previous design's three separate root-span traces linked only by a shared attribute. Verify the actual parent/child rendering empirically (the synthetic parent has no real recorded span, so Tempo may show these as top-level spans within the one trace rather than a clean tree — confirm what actually happens, don't assume a specific visual shape). --- ### Task 11 — `authmtx`: read `SERVICE_VERSION`/`PLEVION_VERSION` from env **Files:** `internal/config/config.go`. - [ ] `Load()` or `validate()`: after YAML unmarshal, override `Service.Version`/ `Service.Environment` from `os.Getenv("SERVICE_VERSION")`/`os.Getenv("PLEVION_VERSION")` when set, falling back to the YAML config's values (currently hardcoded literals `"latest"`/`"local"` in `services/mediamtx/base/authmtx.yml`) otherwise — these env vars are already injected into this exact container via the existing Phase 1 `configMapKeyRef` mechanism (confirmed directly, no `plevion-k8s` change needed for this task). **Acceptance:** - [ ] `go test -race ./...` passes (add a config-loading test covering the env-override case if one doesn't already exist for this function). - [ ] `UserAgent()`'s output reflects the real injected version, not `"latest"`, when run in the actual Deployment. --- ### Task 12 — `frontend-landing` / `frontend-admin`: complete resource-attribute tagging **Files:** `instrumentation.node.ts` (both repos, same shape), `package.json` (both). - [ ] Decide (Plan-time flag, not silently resolved): bump `@opentelemetry/semantic-conventions` past `^1.25.1` to get `ATTR_DEPLOYMENT_ENVIRONMENT_NAME` from the stable entrypoint, import it from the `/incubating` subpath at the current pin, or use the raw attribute-key string literal (`"deployment.environment.name"`) directly without the constant. Pick one, apply identically to both repos (they share the exact same pinned version and `Resource({...})` shape). - [ ] Add `service.version` (from `process.env.SERVICE_VERSION` — confirm this env var is actually injected into these two Deployments the same way it is everywhere else, add if missing) and `deployment.environment.name` (from `process.env.NODE_ENV` or a dedicated env var — decide which is more accurate, don't assume `NODE_ENV` maps cleanly onto `local`/`dev`) to the existing `Resource({...})` call in both files. **Acceptance:** - [ ] Both repos: `npm run lint` + `npm run build` (+ `npm run test` for `frontend-admin`, which has a test suite unlike `frontend-landing`) pass. - [ ] A real trace from either frontend shows `service.version`/ `deployment.environment.name` as resource attributes in Tempo. --- ### Checkpoint 2 (final, before merging any of Tasks 8-12's PRs) Confirm the `plevion-k8s` PR (Tasks 1-6.5) is merged and live first — every other PR's live verification step depends on a real, authenticated collector existing, including the external Tailscale ACL grant (Task 2b's checkpoint). Merge order recommendation: `plevion-k8s` (Tasks 1-6.5) → `edge-controller` (Task 8) → `orchestrator` (Task 9) → `authmtx` (Tasks 10-11) → frontends (Task 12), roughly by dependency depth, though `authmtx`/frontends have no hard ordering dependency on each other or on `edge-controller`/`orchestrator` and can land in any order relative to those two. ### Checkpoint 3 (whole-plan verification) - [ ] Every service in the corrected audit table shows up as a distinct `service.name` in Tempo with correct `service.version`/`deployment.environment.name` resource attributes. - [ ] `mediamtx`'s Prometheus metrics are scraped and visible in Grafana. - [ ] A real MediaMTX auth/ready/notready cycle for one connection produces three `authmtx` spans **sharing one Tempo trace ID** (Feature Design 3b's shared-trace correlation via `$MTX_SOURCE_ID`, not the earlier attribute-based-only design) — confirm by opening that trace ID in Tempo and seeing all three; also confirm the fallback path (malformed/missing `source_id`) degrades to a normal root span rather than erroring. - [ ] The tailnet OTLP path (Task 2b) rejects a token-less export and accepts a correctly-tokened one; the Tailscale ACL grant (external, manual) is confirmed done. - [ ] Monitoring's preflight/postflight checks (Task 6.5) fire correctly: a healthy run reports a confirmed trace, and neither block does anything on a plain `./ansible/bootstrap.sh` run without `--with-monitoring`. - [ ] A real request through Kong shows a Kong-attributed span. - [ ] Grafana's Loki-to-Tempo log click-through still works after Task 6. - [ ] `./ansible/bootstrap.sh` (no flag) still works exactly as before this plan — monitoring stays off, no regression to the default fast-feedback-loop path. ## Boundaries - **Always**: read the actual current k8s Deployment/Job `command:`/`env:` before claiming a service is or isn't instrumented — this plan exists because a prior draft trusted `pyproject.toml`/`CLAUDE.md` claims instead and got it wrong in both directions. - **Ask first**: confirm the Tailscale ACL grant (Task 2b's external checkpoint) is actually applied before treating edge-device telemetry as working — this repo's own tooling cannot verify it; treat the bearer token the same as every other real secret in this repo (SOPS+age, never logged, never committed decrypted). - **Never**: bump `monitoring/overlays/local/values/otel.yml`'s pinned collector `image.tag` (`0.105.0`) while touching this file for Task 2, 2b, or 6 — it's load-bearing for the deprecated-but-still-required `loki` log exporter at that version; add the `opentelemetry-instrument` wrapper *before* `uv run` in `sink-cqrs-worker` (Task 3) — it must go inside, wrapping the Python process `uv` launches, not `uv` itself; add `auth:`/`bearertokenauth` to the *existing* `otlp` receiver (Task 2b must use a second, dedicated receiver instead, or every already- wired in-cluster service starts failing to export).