Status: Plan phase in progress — revised twice. First: an independent Opus 5 Spec
Revision pass found two factual errors in the original "empirical" audit (one of them
undermining the spec's largest Feature Design) plus several real feasibility gaps.
Second: a Plan-phase investigation (triggered by user follow-up questions on the
tailnet-auth gap, the monitoring-workload verification gap, and whether MediaMTX's
own custom image could do better than attribute-based trace correlation) resolved all
three with concrete, source-grounded designs — see Feature Design 2c (new) and the
updated Feature Design 3b. See "Revision Changelog" at the end for exactly what
changed and why, across both passes.
Written against: plevion-k8s main @ dbe1c31 (Phase 2 core merge), auth/orchestrator/edge-controller/sink main tips as of 2026-08-01, authmtx main
@ 6213123, mediamtx main @ 4119e26, kong main @ c9e2020, frontend-admin/frontend-landing main tips as of 2026-08-01.
Close out the last open piece of plevion-docs/ROADMAP.md Priority 2. ROADMAP.md's
telemetry section has four bullets; this spec covers all four, though bullets 1 and 4
(log readability / collector pipeline verbosity) turn out to be directly coupled to
bullet 3 (trace correlation) rather than separable — see Feature Design 6's "log noise
vs. trace correlation" subsection, forced into scope by Spec Revision finding 14.
Concretely: deploy the monitoring stack at all (it currently never is), close the
handful of real per-service instrumentation gaps that remain once the (mostly
already-wired) Python services are correctly audited, get both Go services
(mediamtx, authmtx) into the telemetry picture with an honest design for what
"trace correlation" actually means for each, enable Kong's tracing (the single entry
point for all traffic), and standardize resource-attribute tagging so results are
queryable by service/version/environment — without breaking the log-to-trace
correlation Grafana already depends on.
This effort was originally framed as "add a way to start the cluster without the
monitoring workload," on the assumption that monitoring is deployed by default today
and needs an opt-out. That assumption is wrong, and it's the root cause of nearly
everything else in this spec, not just a convenience gap.
ansible/roles/infrastructure/tasks/main.yml applies infrastructure/overlays/{env}
only (MetalLB, cert-manager, Tailscale operator).
ansible/roles/services/tasks/main.yml applies services/overlays/{env}[-with- frontend] only.
No role applies monitoring/overlays/{env} or environments/local/ as a whole.
Confirmed exhaustively: ansible/roles/ contains exactly cluster_bootstrap, frontend_docker, infrastructure, migrations, postflight, preflight, secrets, services — none references monitoring/.
This means the OTel Collector is never running on local, ever, regardless of any
other change in this spec. Every currently-instrumented service (see the corrected
audit below — this is most of them) is exporting into a void: OTEL_EXPORTER_OTLP_ENDPOINT: http://$(HOST_IP):4317 with nothing listening on that port. Feature Design 1 is not
one item among several — it's the blocking prerequisite for every other Feature Design
in this spec to be verifiable at all.
Confirmed with the user: follow the existing --with-frontend precedent exactly
(opt-in, off by default) — new flag is --with-monitoring. See Feature Design 1 for a
real consequence of this default that the original draft missed.
The original draft claimed auth/orchestrator/sink/edge-controller all have "OTel
packages listed, zero runtime wiring." This was true only for edge-controller.
The other three are wired via command: ["opentelemetry-instrument", ...] overrides in
their k8s Deployments/Jobs — a location the original audit didn't check (it looked at
each repo's own docker/ directory, which has no CMD/ENTRYPOINT at all; the
runtime command lives entirely in plevion-k8s/services/*/base/*.yml).
Service / workload | Wired? |
| Gap |
|---|---|---|---|
| ✅ | ❌ | Add |
| ✅ | ❌ | Add |
| ✅ | ❌ | Add |
| ✅ | ✅ | none |
| ✅ | ✅ | none |
| ❌ | n/a | Add wrapper |
| ✅ | ✅ | none |
| ❌ | n/a | Add wrapper |
| ✅ | ✅ | none |
| ⚠️ | ✅ | Wrapper must go inside |
| ❌ | n/a | Zero OTel deps anywhere in the repo at all ( |
| ✅ real | ✅ ( | Only |
| ✅ same | ✅ ( | Same gap |
| N/A (upstream binary) | — | See Feature Design 3a |
| ❌ zero telemetry | — | See Feature Design 4. |
| ❌ zero telemetry configured | — | See Feature Design 5 |
Even the fully-wired services (most of auth/orchestrator/sink) emit nothing
useful today — not because they're unwired, but because Feature Design 1 means there's
nothing listening on the other end. Fixing FD1 first, then verifying what already
flows, then closing the small residual per-workload gap list above, is the real shape
of this work — not "instrument everything from scratch."
Monitoring deployment: new opt-in --with-monitoring flag. Off by default,
matching --with-frontend's precedent exactly. Revised: see Feature Design 1 for
a real consequence of "off by default" the original draft didn't account for
(every wired service's OTLP exporter connection-refused-loops against a collector
that doesn't exist, in the default case).
Python services get real instrumentation, not just an audit — revised. Most
of auth/orchestrator/sink are already wired (see corrected audit above). The
real scope is the specific gap list in that table, plus edge-controller, which
genuinely does need real instrumentation added from zero.
edge-controller has two real deployment modes, both in scope — unchanged from
the original framing, but see Feature Design 2b for corrected mechanics: the
in-cluster case is not "trivial" (it needs the same wrapper + env vars as every
other service, currently has neither), and the tailnet case needs a new Service
resource that doesn't exist yet, not just a flag.
mediamtx needs trace correlation, not just metrics — revised twice now: once to
attribute-based correlation, once further to genuine shared-trace correlation.
MediaMTX's authHTTPAddress auth-decision call genuinely has zero header/context
customization surface — confirmed by reading MediaMTX's own source at the pinned
tag (internal/auth/manager.go's authenticateHTTP does a hardcodedhttpClient.Post with a plain &http.Transport{}), so there is still no way to
carry an inbound traceparent from the triggering RTSP/WebRTC/SRT connection (which
has no HTTP headers to propagate from in the first place) — that part of the
original correction stands.
But a second Plan-phase investigation (triggered by the question "can the custommediamtx image itself inject tracing?") found something better than root-span-plus-
attributes: MediaMTX already generates one stable UUID per connection, reused
unchanged as the /auth call's id field, $MTX_SOURCE_ID in the runOnReady/runOnNotReady hook environment, and the APISourceDescribe ID — confirmed by
tracing c.uuid through MediaMTX's own internal/servers/rtmp/conn.go andinternal/hooks/on_ready.go source at the pinned tag, not assumed. authmtx'shandler/auth.go already decodes this ID from /auth's JSON body today; it's simply
unused. Passing the same ID to the runOnReady/runOnNotReady hooks too (a one-linemediamtx.yml query-param addition — no image/binary change needed at all, despite
the question assuming a custom wrapper binary would be required) lets authmtx
derive a deterministic trace ID from that UUID (uuid.UUID and OTel'strace.TraceID are both [16]byte — a direct reinterpretation, no hashing) for all
three hook calls. Result: auth/ready/notready for one MediaMTX connection
become three spans sharing one real, queryable trace ID in Tempo — genuine
same-trace correlation for that connection's full lifecycle, not attribute-search
correlation across three disconnected traces. The synthetic parent isn't a real
recorded span, so Tempo will likely render these as top-level spans within that one
trace rather than a clean parent→child tree (confirm exact rendering empirically
during Build) — cosmetically imperfect, but a real, substantive upgrade. See Feature
Design 3b for the full mechanism and file-level detail.
What still isn't achievable, unchanged from the first correction: this shared
trace still doesn't nest under sink/orchestrator's own distributed trace for the
same stream (no inbound trace context exists to link them) — cross-referencing those
still happens by looking up the shared object ID in Tempo, not by trace nesting.
authmtx gets real OTel Go SDK instrumentation — unchanged, but see Feature
Design 4 for an honest dependency-cost accounting (adding the gRPC-based OTLP
exporter pulls in dozens of transitive modules, not "small and well-scoped" as
originally stated — mitigated by using the HTTP exporter instead, see below).
--with-monitoring flag + new monitoring ansible roleFiles: ansible/bootstrap.sh, ansible/site.yml, newansible/roles/monitoring/tasks/main.yml, monitoring/overlays/local/values/otel.yml
(new Service resource, see below), monitoring/overlays/dev/ (new — see dev-guard
below).
Mirror with_frontend exactly:
bootstrap.sh: new --with-monitoring flag, WITH_MONITORING="false" default,
passed through as -e "with_monitoring=${WITH_MONITORING}".
site.yml: new with_monitoring: false var default; new monitoring role added to
the roles: list, when: with_monitoring | bool, placed after infrastructure and
before migrations/services (no race found against either — monitoring targets
its own namespace and nothing in migrations/services/postflight reads from it
at apply time).
New role applies monitoring/overlays/{env} and waits for the OTel Collector
DaemonSet to have at least one Ready pod and confirms the readiness check
targets the actual path services use (hostIP:4317, not just "pod is Running" —
see below for why this distinction matters) before returning.
--env=dev guard, new in this revision: monitoring/overlays/ currently has only
a local/ subdirectory. site.yml's Validate env assert accepts env in ['local', 'dev'], so --with-monitoring --env=dev is a reachable flag combination that would
fail on a missing kustomize target. The monitoring role must either assertmonitoring/overlays/{env} exists before attempting to build it (clear failure
message, not a raw kustomize error) or dev parity needs to be built as part of this
phase — decide in Plan; at minimum this is a guarded failure mode, not silently
ignorable the way the original draft's Open Question framed it.
In-cluster reachability mechanism — corrected, new in this revision: every
currently-wired service reaches the collector via $(HOST_IP):4317 (a fieldRef: status.hostIP + the DaemonSet's hostPort: 4317), not a Kubernetes Service. This
already works today for any pod on a node running a collector instance and needs no
new Service for the in-cluster case. A new Service is needed, but only for Feature
Design 2b's tailnet-exposure case (an edge device has no concept of "the node's host
IP" the way an in-cluster pod does) — see FD2b for why this must be a manually-authored
Service (not the chart's own service.enabled toggle, which defaults tointernalTrafficPolicy: Local in DaemonSet mode and would only route to whichever
collector replica happens to share a node with the Tailscale proxy pod).
Default-off consequence — new in this revision: with monitoring off by default (the
confirmed decision), every already-wired service's OTLP exporter will connection-refuse-loop
against a nonexistent collector on every plain bootstrap.sh run. OTel SDK exporters are
designed to fail this way non-fatally (retry-with-backoff in a background thread, not a
startup-blocking or request-blocking failure) — but this is worth confirming empirically
in Plan/Build rather than assumed, and is plausibly a real contributor to the log-verbosity
complaint this same ROADMAP item flags (a connection-refused retry loop is exactly the
kind of noise a developer would want stripped, and stripping it without understanding the
cause could look like "fixing" something that's actually inherent to the opt-in design).
Document this explicitly wherever exporter behavior is discussed with developers, rather
than leaving it as a silent surprise.
Version-pin boundary, new in this revision: otel.yml pins image.tag: "0.105.0"
while presumably newer chart versions exist — this pin is load-bearing, not stale
by oversight: the collector's logs pipeline uses the loki exporter, which upstream
deprecated in mid-2024 (scheduled for removal, replacement is otlphttp). Do not
"helpfully" bump this image tag as a drive-by while touching this file for the new
Service resource or the prometheus receiver (Feature Design 3a) — that would silently
break the entire logs pipeline. Call this out explicitly as a Never in Boundaries.
Transition-away case (re-running without --with-monitoring after previously
running with it): whether to tear down monitoring or leave existing Loki/Tempo data in
place remains an Open Question — not resolved by this spec.
Files: per the corrected audit table above — services/orchestrator/base/crons/ {reap-connections,check-components-updates}-cron.yml, services/sink/base/gateways/ broadcast/deployment.yml (add opentelemetry-instrument wrapper to each); services/ sink/base/workers/sink-cqrs-worker.yml (wrapper must wrap inside the uv run
invocation: uv run opentelemetry-instrument python manage.py cqrs_consume, not
before uv run, or the instrumentation never actually runs under uv's resolved
environment); services/auth/base/deployment.yml, services/auth/base/gateways/ events/deployment.yml, services/orchestrator/base/deployment.yml (addOTEL_SERVICE_NAME, currently missing — these three currently report asunknown_service if they ever did export successfully).
This is a small, enumerable list — not "instrument every Python service from scratch."
Verify opentelemetry-bootstrap (the distro's auto-instrumentation-detection step) has
actually run for each repo's dependency set before assuming the already-wiredopentelemetry-instrument calls have anything registered to instrument — check each
repo's build step, don't assume.
edge-controller's two deployment modesFiles: edge-controller's pyproject.toml (new OTel deps — the full set, since
none exist today), edge-controller's Dockerfile/entrypoint (wrapper), plevion-k8s/ services/edge-controller/base/deployment.yml (currently has neither the wrapper nor
any OTEL_* env vars — not the "trivial" case the original draft claimed, this
needs the same treatment as every other service in Feature Design 2a, from zero), plus:
orchestrator/src/apps/components/ansible/templates/controllers.py — theEDGE_CONTROLLER_TMPL's environment: block already has exactly the right
precedent (SERVICE_VERSION: {{ edge_controller_version }}, confirmed by direct
read), so add OTEL_EXPORTER_OTLP_ENDPOINT/OTEL_SERVICE_NAME there the same way.
Also add the opentelemetry-instrument wrapper to this same template'scommand: (currently celery -A settings worker -l info ... — env vars alone do
nothing without the wrapper, same as every gap in Feature Design 2a). Name the new
Ansible-rendering context value explicitly in Plan — the call site
(orchestrator/src/apps/components/ansible/playbooks/components.py) sources
everything else from settings.* (e.g. settings.API_URL), so the new OTLP-endpoint
value should follow that same pattern (e.g. settings.OTEL_COLLECTOR_TAILNET_ENDPOINT),
not be hardcoded in the template.
New, corrected in this revision: a manually-authored Service resource (not the
Helm chart's own service.enabled toggle — see FD1's note on why that defaults tointernalTrafficPolicy: Local in DaemonSet mode) selecting the OTel Collector
DaemonSet's pods, with default (cluster-wide) traffic policy so it round-robins
across whichever collector replica is reachable — edge devices don't need pod
affinity to a specific node the way hostIP-based in-cluster services do, they just
need one stable endpoint. This Service is then what gets exposed to the tailnet.
Tailnet exposure mechanism — narrowed in this revision: the original draft listed
three options (Ingress, Service annotation, Connector/ProxyGroup CRDs) without
checking which actually applies. Verified against Tailscale's own operator docs:ingressClassName: tailscale is Tailscale Serve — built for HTTP(S) web
apps/APIs with TLS termination, the wrong shape for raw OTLP gRPC (h2c, no
TLS/ALPN needed). Connector is a subnet-router/exit-node CRD, not a
single-Service-exposure mechanism. The correct mechanism is thetailscale.com/expose: "true" annotation (or loadBalancerClass: tailscale) on the
new Service above — this operates via iptables/nftables DNAT below the
application layer, protocol-agnostic, confirmed to work for arbitrary TCP/gRPC
traffic.
Authentication — resolved via a dedicated Plan-phase investigation, defense in
depth, both layers real and concrete, not a single either/or pick:
Application layer: a second, dedicated OTLP receiver instance
(otlp/tailnet, new ports — not the existing :4317/:4318 every in-cluster
service already uses) protected by the bearertokenauth extension — confirmed
compiled into this repo's exact pinned collector image (otelcol-contribv0.105.0) by reading that release's own component manifest. A shared token,
stored the same way every other secret in this repo is (newsecrets/otel-collector.enc.env, SOPS+age, decrypted intomonitoring/overlays/local/.secrets/ by the existing secrets role's pattern),
checked against the standard Authorization: Bearer <token> header — which is
exactly what the OTel SDK's own OTEL_EXPORTER_OTLP_HEADERS env var is designed
to set, so the edge-device side (Feature Design 2b's Ansible template) needs no
new mechanism, just one more env var line next to OTEL_EXPORTER_OTLP_ENDPOINT.
A second receiver rather than adding auth: to the existing one, specifically
so every already-wired in-cluster service is completely unaffected — bolting auth
onto the shared receiver would 401 all of them unless every Deployment also
threaded the token through, real scope creep this design avoids entirely.
Network layer: a Tailscale ACL tag (tailscale.com/tags: "tag:otel-collector",
same service.annotations block as the expose annotation) restricting which
tagged devices can reach the exposed Service at all — this repo already has live
precedent for tag-based device identity (bluespot advertisestag:remote-controllers on every field device's provisioning, confirmed by
direct read of bluespot/src/apps/registration/services/tailscale.py), just not
yet an ACL grant using it for anything. The actual grant is a manual step in the
Tailscale admin console's policy file (this repo has no checked-in ACL policy at
all today, confirmed by search — same as how the operator's own OAuth client
credentials are already provisioned outside this repo's git history), not
something a Kustomize/Ansible change can apply — state this plainly as an
external, human setup step wherever this is documented, not something CI/kubectl kustomize can verify.
See Feature Design 2c for the concrete file-level design.
Files: monitoring/overlays/local/values/otel.yml (new otlp/tailnet receiver +bearertokenauth extension + extraEnvs + new ports: entries, alongside the
existing otlp receiver, unchanged), monitoring/overlays/local/kustomization.yml
(new secretGenerator block — this file has none today, confirmed by direct read),ansible/roles/secrets/tasks/main.yml (new decrypt step, mirroring the existingtailscale.enc.env/orchestrator.enc.env entries exactly), newsecrets/otel-collector.enc.env; orchestrator/src/settings/common.py (newOTEL_COLLECTOR_BEARER_TOKEN setting, same pattern as the newOTEL_COLLECTOR_TAILNET_ENDPOINT setting from Feature Design 2b),orchestrator/src/apps/components/ansible/templates/controllers.py (newOTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer {{ otel_collector_bearer_token }}"
line in EDGE_CONTROLLER_TMPL's environment: block).
Config shape (verified against config/configauth's README at the matching
opentelemetry-collector-contrib core tag, and bearertokenauthextension's own source
— server-side Authenticate() checks an Authorization/gRPC-metadata value against"Bearer <token>", exactly the standard shape OTEL_EXPORTER_OTLP_HEADERS sets):
New ports: entries for 4319/4320 need containerPort but deliberately nohostPort — this path is meant to go only through Feature Design 2b's Service, not
the existing hostPort mechanism every in-cluster service already uses.
mediamtx: enable native Prometheus metricsFiles: plevion-k8s/services/mediamtx/base/mediamtx.yml, deployment.yml (port
9998), monitoring/overlays/local/values/prometheus.yml (new scrape job — not the
OTel Collector's own prometheus receiver, corrected below).
Add metrics: yes / metricsAddress: :9998 to mediamtx.yml — verified directly
against upstream MediaMTX v1.19.1's own mediamtx.yml reference (the exact version
this repo's Dockerfile pins): real, documented, Prometheus-compatible, no custom code
needed.
Scrape mechanism — corrected in this revision: the original draft's "likely right
call" (route mediamtx's metrics through the OTel Collector's own prometheus receiver,
for consistency with everything else going through one collector) is wrong for this
specific case — the collector runs as a DaemonSet, so every node's collector
instance would independently scrape mediamtx's single pod, producing duplicate metric
series on any multi-node cluster (invisible on today's single-node microk8s, a real bug
the moment dev becomes a real multi-node target). Use a direct Prometheus scrape job
instead (prometheus.yml's existing extraScrapeConfigs pattern already has one job
for otel-collector pods specifically — add a second, equivalently scoped, job
targeting mediamtx's pod label on port 9998).
Defensive config addition, new in this revision: mediamtx.yml doesn't currently
set authHTTPExclude, so it inherits upstream's default (action: api, action: metrics, action: pprof all excluded from the auth-hook path) — which is why enablingmetrics: yes is safe today without also routing metrics scrapes through authmtx's/auth handler (which has no route for that and would break). This is currently an
implicit dependency on an upstream default, not a decision this repo made explicitly.
Add authHTTPExclude explicitly, listing api/metrics/pprof, so it's
self-documenting and doesn't silently break if anyone later sets authHTTPExclude for
an unrelated reason and overwrites the inherited default.
mediamtx: shared-trace correlation via $MTX_SOURCE_IDCorrected twice now — see Confirmed Scope Decision 4 above for the full history
(root-span-plus-attributes, then upgraded to genuine shared-trace correlation via a
second Plan-phase investigation). Concrete mechanism, grounded in MediaMTX's own
source at the pinned tag (v1.19.1), not assumed:
services/mediamtx/base/mediamtx.yml: add &source_id=$MTX_SOURCE_ID to both
existing runOnReady/runOnNotReady wget URLs (currently pass path/query/source_type only). $MTX_SOURCE_ID is the same UUID MediaMTX already sends on
every /auth call's JSON body (id field) — confirmed identical across a
connection's /auth → runOnReady → runOnNotReady lifecycle by tracing c.uuid
through internal/servers/rtmp/conn.go (generated once, at connection-object
creation) and internal/hooks/on_ready.go (the same closure/env reused for both
hook firings). One-line config change, no image/binary change.
authmtx: add github.com/google/uuid (tiny, no transitive deps). In each
handler (auth.go already decodes the ID from the JSON body; ready.go/stop.go
need to read the new source_id query param), parse the UUID and construct a
deterministic trace.SpanContext from its 16 bytes (uuid.UUID andtrace.TraceID are both [16]byte — direct reinterpretation), inject it as a
remote parent context, then start each handler's real span from that context. This
is handler-level work, not something the generic otelhttp.NewHandler(mux, ...)
middleware from Feature Design 4 can do on its own (that middleware runs before the
request body/query params are parsed) — additive to it, not a replacement.
Result: auth/ready/notready for one MediaMTX connection become three spans
sharing one real, queryable Tempo trace ID — open by trace ID, see the full
lifecycle. The synthetic parent span ID won't correspond to a real recorded span, so
Tempo will likely render these as top-level spans within that one trace rather than a
clean parent→child tree — verify the exact rendering empirically during Build, don't
assume a particular visual shape.
Still not achievable, unchanged: no nesting under sink/orchestrator's own
trace for the same stream (no inbound trace context exists to link them at the
RTSP/WebRTC/SRT layer) — cross-referencing those still happens by looking up the
shared object ID in Tempo.
authmtx: OTel Go SDK instrumentationFiles: go.mod (new deps), cmd/authmtx/main.go (SDK init/shutdown),internal/server/server.go (wrap the mux with otelhttp.NewHandler),internal/upstream/client.go (wrap the outbound HTTP client's Transport withotelhttp.NewTransport), services/mediamtx/base/authmtx.yml (stop hardcodingservice.version/service.environment, read from env instead — see below).
Dependency-cost correction, new in this revision: authmtx/go.mod today has
exactly one dependency (gopkg.in/yaml.v3). The original draft called adding OTel
instrumentation "small, well-scoped... not a broad dependency expansion" — this
understates it. The gRPC-based exporter (otlptracegrpc) pulls in google.golang.org/ grpc, google.golang.org/protobuf, genproto, and several more transitive modules —
dozens of new entries in go.sum. Use otlptracehttp instead — the collector
already has its OTLP HTTP receiver enabled (:4318, confirmed in otel.yml) alongside
gRPC, and the HTTP exporter avoids pulling in the entire gRPC/protobuf tree. State the
real module-count tradeoff honestly in Plan rather than calling it "small."
internal/server/server.go's existing structure (mux → recoveryMiddleware →http.Server) makes this a clean wrap: otelhttp.NewHandler(wrapped, "authmtx")
around the existing middleware chain. internal/upstream/client.go's outbound call to
Kong gets otelhttp.NewTransport on its http.Client.Transport.
Resource attributes, corrected in this revision — this was Open Question 6 in the
original draft, now answered by direct verification, not left open:services/mediamtx/base/authmtx.yml already has service: {name: "authmtx", version: "latest", environment: "local"} — name is fine, but version/environment are
hardcoded literals, not read from Phase 1's SERVICE_VERSION/PLEVION_VERSION env
vars (which are already injected into this container via configMapKeyRef: {name: plevion-platform-version, key: authmtx} — confirmed directly, no gap there).
The actual work is in internal/config: read SERVICE_VERSION/PLEVION_VERSION from
env and use them instead of the config file's hardcoded strings.
kong: enable tracing — corrected target, this revisionThe original draft's Files line (kong.yml) was wrong and would have shipped a
no-op. Verified: plevion-k8s deploys Kong via env.database: "off" +ingressController.enabled: true (services/kong/base/values.yml) — config comes from
Kubernetes CRs (KongClusterPlugin, following the exact precedent already in this repo
at services/overlays/local/auth-plugin.yml), not a declarative file. The kong
repo's own kong/kong.yml is dead, stale docker-compose-era config
(url: "http://mediamtx:8889", auth_url: "http://auth-service:8081" — neither
resolves in this cluster) that the Kong image doesn't even COPY — it's not consumed
anywhere in plevion-k8s.
Files (corrected): new services/overlays/local/otel-plugin.yml (aKongClusterPlugin, plugin: opentelemetry, mirroring auth-plugin.yml's shape —config.endpoint pointed at the collector's OTLP HTTP receiver, :4318/v1/traces —
verify in Plan whether this Kong version's plugin schema wants HTTP or gRPC, don't
assume); services/kong/base/values.yml's env: block (new: KONG_TRACING_INSTRUMENTATIONS
and KONG_TRACING_SAMPLING_RATE — required, confirmed against Kong's own plugin
docs: "Kong Gateway creates a top-level span for each request... whentracing_instrumentations is enabled" — the plugin alone, without these, produces no
spans at all). Also verify config.propagation vs. the deprecated config.header_type
against Kong 3.5's actual accepted schema (header_type is deprecated as of a version
at or before this repo's pin — confirm exactly).
Rationale, corrected in this revision: the original draft claimed "without this,
every downstream service's traces start fresh at Kong regardless of how well
instrumented they are individually" — false. Kong forwards inbound request headers
to upstreams by default; a traceparent header from a frontend's instrumented request
already reaches Django's own instrumentation whether or not this plugin is enabled.
What the plugin actually adds is a Kong-level span in the trace (visibility into
Kong's own request-handling time, not the thing that makes propagation possible at
all). Enabling it is still the right call — Kong is the single entry point for all/public/v1/* traffic and a Kong-side span is genuinely useful — just state the real
reason, not an invented one.
Files: every service touched above, frontend-landing/frontend-admin'sinstrumentation.node.ts.
service.name — already correct everywhere it's set (plevion-landing,plevion-admin; the rest come from closing Feature Design 2a/2b's gaps).
service.version — tie to the existing SERVICE_VERSION env var (Phase 1'splevion-platform-version ConfigMap injection) rather than inventing a parallel
mechanism — already the right call, unchanged from the original draft.
deployment.environment.name — new caveat, this revision: frontend-landing/frontend-admin pin @opentelemetry/semantic-conventions@^1.25.1.ATTR_DEPLOYMENT_ENVIRONMENT_NAME is an incubating export (available only via
the /incubating subpath) added after that pin — not importable from the stable
entrypoint as currently pinned. Plan needs to decide: bump the semantic-conventions
package, import from the incubating subpath, or use the raw attribute-key string
directly rather than the constant. Don't assume the two-key Resource({...}) edit
the original draft implied is a trivial one-liner.
Log noise vs. trace correlation — new section, this revision, resolving Spec
Revision finding 14 (the original draft dropped this entirely): monitoring/overlays/ local/values/grafana.yml's Loki datasource does log-to-trace linking viaderivedFields: matcherRegex: "trace_id=(\\\\w+)" — this depends onopentelemetry-instrumentation-logging injecting trace_id= into every log line,
which is exactly the kind of "OTel-added prefix/suffix noise" ROADMAP.md's log-
readability bullet wants stripped. These two ROADMAP asks are in direct tension and
must be resolved together, not separately:
Resolution: don't disable the logging instrumentation package (that would break
Grafana's derived-field link entirely). Instead, address log noise in the OTel
Collector's own log-processing pipeline (a transform processor trimming
cosmetic/redundant fields from the rendered log line) while leaving thetrace_id=-bearing structure Grafana depends on intact. This needs to be verified
against the collector's actual current log-processing config in Plan — don't assume a
specific processor without checking what's already there.
Logs-via-OTLP, explicitly out of scope, new in this revision: the collector'slogs pipeline currently has only a filelog receiver — no otlp receiver (the
config file's own comment notes this: "If you wanted to receive OTLP logs from the
app, add 'otlp' here"). Any log records a service's OTel SDK exports directly over
OTLP are silently dropped today; all logs actually arrive via node-file scraping
instead, which carries k8s.* resource attributes, not service.* ones. This spec
does not propose adding an otlp logs receiver — ROADMAP.md's ask is aboutkubectl logs/Loki readability, which the existing filelog path already serves.
State this as a deliberate boundary, not an oversight, if it comes up in Plan.
local primarily. dev parity is explicitly not attempted by this spec beyond the
guard in Feature Design 1 (fail clearly, don't silently break, if --with-monitoring --env=dev is invoked before monitoring/overlays/dev/ exists) — full dev telemetry
parity remains a separate, not-yet-scoped follow-up per ROADMAP.md's own "dev is
partial" item.
opentelemetry-bootstrap has run (or needs to be added) per-repo for the
already-wired services — verify, don't assume, per Feature Design 2a.
The OTel Collector's existing otlp receiver (:4317 grpc, :4318 http) needs no
reconfiguration to accept traffic from newly-instrumented services.
OTLP exporters in the Python/Go SDKs fail non-fatally (background retry, not
startup/request-blocking) when the collector is unreachable — assumed based on
standard SDK behavior, not yet empirically confirmed against this repo's actual
configuration; confirm in Plan/Build (see Feature Design 1's default-off note).
Offline-verifiable: ansible-playbook site.yml --syntax-check after Feature Design
1's new role; yamllint/kubectl kustomize --enable-helm after every manifest
change.
Live-verifiable against the existing local microk8s cluster, no new credentials
needed: every piece here, once --with-monitoring exists.
authmtx/Kong changes: go test -race ./... / existing manual docker compose up
verification steps, plus a live check that a real span appears in Tempo for an actual
MediaMTX auth/ready/stop cycle and a real Kong-routed request.
Always: reuse the existing SERVICE_VERSION/Phase-1-ConfigMap mechanism forservice.version rather than inventing a parallel one; follow the --with-frontend
precedent's exact shape for --with-monitoring; verify Kong's plugin schema and
MediaMTX's config keys against their real docs/source before writing config, not from
memory or from what a prior draft of this spec assumed.
Ask first: whether re-running bootstrap.sh without --with-monitoring after a
prior --with-monitoring run should tear down the monitoring stack or leave it.
(The OTLP-receiver-on-the-tailnet authentication gap that was previously listed here
is now resolved — see Feature Design 2c — but the Tailscale ACL grant itself is a
manual, external admin-console step no file in this repo can apply or verify; treat
that step, specifically, with the same "ask first"/confirm-before-relying-on-it care
this repo's CLAUDE.md already applies to networking/exposure decisions.)
Never: treat "the OTel packages are listed in pyproject.toml" as equivalent to
"this service emits telemetry" — get this from reading the actual k8s Deployment
command/env, not the repo's own dependency file or its CLAUDE.md's claims (both
were wrong, in different directions, for services audited during this spec's
Revision pass); bump monitoring/overlays/local/values/otel.yml's pinned collector
image tag as a drive-by while touching this file for other reasons — the pin is
load-bearing for the (deprecated but still-required-at-this-version) loki log
exporter, see Feature Design 1.
Feature Design 1's transition-away behavior (teardown vs. leave monitoring data).
Feature Design 2b's OTLP-receiver-on-tailnet authentication design — resolved,
see Feature Design 2c. The Tailscale ACL grant's manual admin-console step remains
an execution-time dependency, not a design question.
Feature Design 6's log-noise-reduction mechanism in the OTel Collector pipeline —
needs to be checked against the pipeline's actual current config, not designed in
the abstract.
Whether a live Tempo-trace-arrival check should be built as a repeatablepreflight/postflight ansible detector or done as a one-time manual check —
resolved: build the repeatable detector, matching Phase 1's self-healing
pattern, per explicit user direction ("a workload needs the same checks as main
services workload"). See the Plan for the concrete preflight/postflight task design
(Tempo's real search API, port 3100 per this repo's actual pinned chart — verified,
not the commonly-cited 3200 default).
Full dev environment telemetry parity — explicitly out of scope for this phase
(see "Environments in scope"), tracked as a separate follow-up.
Independent Opus 5 Spec Revision pass found:
Factual error, blocking: the original "empirical audit" claimed auth/orchestrator/sink had zero runtime OTel wiring — false; all three are wired viaopentelemetry-instrument command overrides in their k8s Deployments, a location the
original audit didn't check. Corrected: rewrote the audit into a per-workload gap
table, most of which already works once Feature Design 1 exists.
Root-cause reframing: "monitoring is never deployed" was correctly identified but
treated as a standalone convenience gap rather than the actual reason nothing works
today. Corrected: stated explicitly as the blocking prerequisite for every other
Feature Design.
FD5 (Kong) factual error, blocking: original target file (kong.yml) is dead,
unused config from a different (docker-compose) deployment context — real Kong
config is via KongClusterPlugin CRDs. Corrected target file and added the requiredKONG_TRACING_INSTRUMENTATIONS/KONG_TRACING_SAMPLING_RATE env vars the plugin
needs to actually produce spans (omitted entirely from the original draft). Also
corrected the stated rationale (Kong already forwards headers by default; the plugin
adds a Kong-level span, doesn't enable propagation).
FD3b/CSD4 design error: "spans nest inside the existing distributed trace" is not
achievable — MediaMTX's hooks are subprocess execs with no trace-context propagation
path. Corrected to attribute-based correlation (root spans + shared ID lookup), and
the internal inconsistency between CSD4's "confirmed answer" framing and FD3b's own
hedge (Open Question 3 in the original draft) is resolved by this correction rather
than left open.
FD2b feasibility gaps: no OTel Collector Service exists for the tailnet-exposure
design to attach to (in-cluster reachability is via hostIP/hostPort, not a
Service); the "trivial" in-cluster edge-controller case actually has no wrapper and
no OTel deps at all; the Tailscale exposure mechanism was left as three
undifferentiated options when only one (Service annotation) is actually correct for
raw OTLP gRPC; no authentication/exposure-risk discussion for a tailnet-reachable
telemetry write endpoint. All corrected/added.
FD1 missing edge case: monitoring/overlays/dev doesn't exist, so--with-monitoring --env=dev (a reachable flag combination) would hard-fail — added
as a guard requirement, not left as a soft Open Question.
FD1 missing consequence: off-by-default monitoring means every already-wired
service's OTLP exporter connection-refuse-loops by default — noted explicitly,
including its plausible connection to the log-verbosity complaint this same ROADMAP
item is trying to fix.
FD3a feasibility error: routing mediamtx's Prometheus metrics through the OTel
Collector's own prometheus receiver would duplicate-scrape under the collector's
DaemonSet deployment mode on any multi-node cluster. Corrected to a direct Prometheus
scrape job. Also added the authHTTPExclude defensive-config note (currently an
implicit, unstated dependency on an upstream default).
FD4 honesty correction: "small, well-scoped, not a broad dependency expansion"
understated adding a gRPC-based OTLP exporter to a one-dependency Go module by a
large margin. Corrected to recommend the HTTP exporter instead (avoids the
gRPC/protobuf transitive tree) and to state the tradeoff honestly either way.
Open Question 6 (original) answered, not left open: SERVICE_VERSION/PLEVION_VERSION already reach authmtx's container via the existing Phase 1
ConfigMap mechanism — confirmed directly. The actual gap is narrower and different
than the question implied: services/mediamtx/base/authmtx.yml hardcodesservice.version/service.environment as literals instead of reading the
already-present env vars.
Scope gap: the original draft addressed only 2 of ROADMAP.md's 4 telemetry
bullets, without stating that narrowing in the Objective. Found (finding 14) that
the other two are directly coupled to what's already in scope (log-noise stripping
would break the Loki-to-Tempo correlation this spec's own Feature Design 6 depends
on) — folded in a scoped resolution rather than leaving the tension unaddressed.
New Boundary items: don't bump otel.yml's pinned collector image tag as a
drive-by (load-bearing for the deprecated-but-required loki exporter at this
version); don't trust either a repo's pyproject.toml/go.mod dependency list or
its own CLAUDE.md claims as evidence of actual runtime telemetry behavior again.
Triggered by three user questions raised while reviewing the Plan: (1) wanting a real
authentication mechanism for the tailnet-exposed OTLP receiver rather than accepting
the exposure as a scoped risk, (2) wanting the monitoring workload to get the same
preflight/postflight self-healing checks every other workload already has, and (3)
asking whether mediamtx's own custom image could inject real tracing rather than
settling for attribute-based correlation. A dedicated investigation (general-purpose
agent, both local source reading and external doc/source fetches) resolved all three:
New Feature Design 2c: a dedicated otlp/tailnet receiver protected by the
collector's bearertokenauth extension (confirmed compiled into the exact pinnedotelcol-contrib v0.105.0 image) plus a Tailscale ACL tag — defense in depth,
doesn't touch the existing in-cluster OTLP path at all. Resolves Feature Design
2b's previously-open authentication gap.
Feature Design 3b upgraded a second time: the MediaMTX-trace-correlation design
(root-span-plus-attributes from the first revision) turned out to be more
conservative than necessary. Tracing MediaMTX's own source at the pinned tag found
a stable per-connection UUID ($MTX_SOURCE_ID) already flows to authmtx today
(unused) and, with a one-line config addition, can also reach the runOnReady/runOnNotReady hooks — enabling a deterministic shared trace ID across a
connection's full auth/ready/notready lifecycle. No image or binary change needed,
contrary to the premise of the question that triggered this investigation.
Open Question 4 (verification mechanism) resolved by explicit user direction,
not left as a spec-level tradeoff: build the repeatable preflight/postflight
detector. The investigation additionally corrected the Tempo query port —
this repo's vendored chart pins 3100, not the commonly-cited 3200 default —
and confirmed the Ansible control node has no direct route to in-cluster*.svc.cluster.local DNS on this topology, so the live check needs to run as an
in-cluster throwaway pod, not a direct ansible.builtin.uri call from the control
node the way the spec's original phrasing implied.