April 27, 2026
Mixed observability for GenAI
OTel substrate + GenAI layer
/ Mixed Observability for GenAI 2026
OTel substrate + GenAI layer
OTel substrate + GenAI layer
Scheme → SOTA stack
Parts 1-2: scheme, architecture.
Parts 3-4: GenAI traces, OTel/Langfuse.
Parts 5-6: market and maturity gaps.
Part 7: SOTA stack and verdict.
OTel as the nervous system + specialized GenAI layer on top
SOTA is a two-layer scheme
OpenTelemetry — substrate
One standard for service telemetry.
Collector: redaction, sampling, fan-out.
GenAI layer on top
Langfuse, MLflow, Phoenix, Datadog.
Prompts, cost, evals, feedback, datasets.
What the chain from user request to AI backend looks like
Mixed prod chain
User/API → app; agent → RAG/model/tool.
Logs/metrics/traces carry context.
Collector: ingest, redact, sample, fan-out.
APM/infra backend + GenAI backend.
Development: active, not yet stable
Model, agent spans; GenAI events/metrics.
gen_ai.operation.name covers chat, retrieval, tools.
gen_ai.agent/tool identify agents/tools.
gen_ai.usage.* provides token metrics.
Decision episode: from user request to final response
Around the business decision, not the model call
Span hierarchy
Root span = business operation.
Child spans = services + GenAI.
Signals
Events/scores: quality, eval, feedback, safety flags.
Metrics: latency, tokens, errors, cost.
OTel-first, Langfuse-as-AI-backend — the most mature pattern
Langfuse is used inside the OTel model
Langfuse as OTel backend
OTLP endpoint: direct or via Collector.
Fan-out: Langfuse + regular APM.
Langfuse SDK v3 over OTel
OTel client with AI conveniences.
Tokens, prompts, scoring, observations.
OTel transport; Langfuse AI UX
Traces, sessions, observations as AI data.
Token/cost tracking with prompt-aware aggregation.
Scores: feedback, judge, programmatic evals.
Prompt management, datasets, experiments, self-hosting.
OTel-first, APM-first, AI-platform-first, Auto-instrumentation
Trade-offs
OTel-first + AI backend: tracing, neutrality, fan-out.
APM-first: single pane, weaker prompts/evals.
AI-platform-first: AI lifecycle + OTel protocol.
Auto-instrumentation: fast start, semantics still vary.
Honest assessment — what's covered, what's partial, what still hurts
Correlation and standardization
HTTP → agent → model/tool trace.
Logs correlate by trace/span IDs.
AI is traced by the same mechanism.
OTel GenAI semconv covers spans/tools/evals.
Decent but not perfect
Tokens and latency are first-class objects.
Inferred pricing misses billing, cache, retries, streaming.
Quality: scores, judges, gen_ai.evaluation.result.
Regressions = usefulness degradation, not exceptions.
Privacy, scale, and tool coexistence
Privacy: prompts/outputs need allowlists/redaction.
Scale: heavier traces; complex tail sampling.
Span filtering breaks trace integrity.
TracerProvider conflicts between OTel tools.
Five principles for the right stack for a mixed prod system
Mixed prod design today
OTel everywhere: traces, metrics, logs.
Collector: redaction, routing, fan-out.
AI-aware backend over OTel.
Root trace = operation; evals near latency.
The industry matured, but not fully
SOTA = OTel protocol/context + AI interpretation/evals.
End-to-end mixed observability is more standard.
Quality and privacy/governance remain open zones.
tellmeabout.tech · ai4sdlc-research.space
OTel, GenAI telemetry, evals
OpenTelemetry documentation and GenAI semantic conventions.
Langfuse, MLflow, Phoenix and OpenInference tracing/evaluation docs.
Datadog LLM Observability; Azure AI Foundry.
Mixed Observability for GenAI
Materials and links are in the "Book Cube" channel
Alexander Polomodov, Technical Director & Fellow, T-Technologies
@Book_Cube