Modern Telemetry with OpenTelemetry Semantic Conventions OTLP and Collector Pipelines

1.3 Distributed Systems Context and Correlation Requirements

Distributed systems need a shared way to describe “what request is this?” and “what work belongs to it?” Without that, metrics become averages of unrelated events, logs become a pile of useful fragments, and traces turn into a set of disconnected timelines.

At the core is context: a small set of identifiers and metadata that travels with the work. Correlation is the discipline of using that context consistently so every signal can be tied back to the same end-to-end flow.

What Correlation Must Achieve

1. End-to-end identity: every hop in a request chain must carry the same trace identity so a trace view can stitch spans into a single story.
2. Causal structure: parent-child relationships must reflect how work was triggered, not just that it happened around the same time.
3. Signal alignment: logs and metrics must include enough context to be filtered or grouped by the same identifiers used in traces.
4. Operational debuggability: when something fails, the context must still be present so you can find the failing component and the exact request.

A practical rule: if you can’t answer “which upstream request caused this downstream action?” using your stored telemetry, correlation is incomplete.

Context Propagation Basics

Propagation means copying context from an incoming request into outgoing calls. In practice, this is usually done through protocol metadata (for example, HTTP headers) and then reattached on the receiving side.

Two identifiers matter most:

• Trace identifier: groups all spans belonging to one end-to-end request.
• Span identifier: identifies the current unit of work so the receiver can set the correct parent.

Additionally, some systems carry baggage: small key-value items that are not strictly required for tracing structure but are useful for debugging and routing decisions.

Correlation Requirements by Signal Type

Traces require correct parent-child linkage. If a service starts work without setting the received context as its parent, the trace becomes fragmented.

Logs require that each log record includes trace identifiers (and optionally span identifiers). This lets you pivot from a trace to the relevant log lines.

Metrics require careful thinking: metrics are often aggregated, so you can’t rely on per-request correlation alone. Instead, correlation is typically achieved by attaching stable dimensions (like service name and operation) and, where feasible, including trace identifiers only for sampled requests or high-signal events.

Common Failure Modes and Their Symptoms

• Missing propagation: downstream spans start new traces, producing multiple traces for one user action.
• Wrong parent assignment: traces show odd trees where child spans appear under unrelated parents.
• Context overwritten: middleware replaces context with a new one, causing logs to point to the wrong span.
• Inconsistent resource attributes: the same service appears with different identity attributes, making aggregation and filtering unreliable.

A quick sanity check is to pick one request, follow it across services, and confirm that trace identifiers remain identical end-to-end.

Example: HTTP Request Across Two Services

Service A receives an HTTP request, creates a server span, and then calls Service B.

• On Service A, the outgoing HTTP request must carry the trace context from the server span.
• On Service B, the incoming request must extract that context and create a child span under the correct parent.

If Service B instead starts a new trace, you’ll see two trace identifiers for what should be one request.

Here’s a minimal conceptual flow:

Client -> Service A -> Service B

Service A
- Create server span for incoming request
- Attach trace context to outgoing HTTP headers
- Log with trace id and span id

Service B
- Extract trace context from headers
- Create child span using extracted parent
- Log with same trace id

Example: Correlating Logs with Traces

When Service B logs an error, the log record should include the trace identifier so you can filter logs to the exact trace. If you also include span identifier, you can narrow further to the specific operation.

A useful operational pattern is to log at the point where the decision is made (for example, after a failed dependency call), not only when the exception bubbles up. That keeps the log aligned with the span that actually represents the failing work.

Validation Checklist

• Identifier continuity: trace identifier stays the same across services.
• Tree correctness: child spans attach to the expected parent.
• Log linkage: log records include trace identifiers for the relevant operation.
• Resource stability: service identity attributes are consistent across deployments.

If these four checks pass for a representative request, correlation is doing its job rather than merely existing.

1.4 Data Lifecycle from Instrumentation to Storage and Query

A telemetry system is only as useful as the path from “something happened” to “someone can ask a question and get an answer.” The lifecycle has five practical stages: instrumentation, collection, transformation, storage, and query. Each stage has its own responsibilities, and each one can quietly break the chain.

Instrumentation Produces Semantically Shaped Events

Instrumentation is where meaning first gets attached to raw activity. For traces, that means creating spans with correct span kinds, consistent trace identifiers, and attributes that match semantic conventions. For metrics, it means choosing instrument types (counter, gauge, histogram) and defining dimensions that won’t explode cardinality. For logs, it means emitting structured fields that can line up with trace and span identifiers.

A simple example: an HTTP handler creates a server span, records the route template and status code, and increments a request counter with the same service identity attributes used elsewhere. If the handler also logs an error, the log includes trace_id and span_id so later queries can join “what failed” with “where it failed.”

Collection Receives Telemetry and Preserves Context

The collector’s job is to receive OTLP data reliably and keep the relationships intact. It accepts data from instrumented services, then routes it into signal-specific pipelines. Context preservation matters: trace context must remain consistent across spans, and resource attributes must remain attached to the correct scope.

A common best practice is to separate pipelines by signal type. That keeps metric transformations from accidentally affecting trace attributes, and it prevents log enrichment rules from changing trace sampling decisions.

Transformation Normalizes Data for Consistent Querying

Transformation is where “it works” becomes “it’s queryable.” Typical steps include:

• Resource normalization: ensure every record has the same service identity fields (service.name, service.namespace, service.instance.id) so grouping is stable.
• Attribute normalization: map inconsistent attribute keys into a single convention, such as converting http.status_code variants into one field.
• Filtering: drop noisy spans or logs early when they are clearly unhelpful, but do it with care so you don’t remove the evidence you later need.
• Metric transformation: adjust units, rename labels, or convert histogram buckets only when you can document the meaning.

Example: if one service emits http.method and another emits request.method, a normalization processor can map both into http.method. Without this, dashboards end up with duplicated filters and confusing “missing” data.

Storage Indexes Data for the Questions You Actually Ask

Storage is not just a database; it’s an indexing strategy. Traces are stored so you can retrieve a trace by trace_id, then navigate spans by time and relationships. Metrics are stored so you can aggregate by time windows and dimensions. Logs are stored so you can filter by fields and correlate with trace identifiers.

A practical rule: design storage expectations around access patterns. If you frequently query “requests by route and status,” then those attributes must be present and consistently named at ingestion time. If you frequently query “error logs for a trace,” then trace_id must be indexed in logs.

Query Joins Signals Through Shared Identifiers

Querying is where the lifecycle either feels coherent or falls apart. The key is shared identifiers and consistent semantics.

• Trace-first: start with a trace_id from an alert or user session, then inspect spans and correlated logs.
• Metrics-first: find an anomaly in metrics, then use service identity and time windows to locate relevant traces and logs.
• Logs-first: filter logs by error fields, then jump to trace_id to see the surrounding span context.

If trace_id is missing from logs, the join becomes a manual scavenger hunt. If resource attributes differ between services, grouping by “service.name” yields fragmented results.

Example End-to-End Flow

A request hits GET /orders/{id}.

1. The service creates a server span with route template and status code, and increments a request counter with method and route dimensions.
2. On failure, it emits a log record with error fields plus trace_id and span_id.
3. The collector receives OTLP data, normalizes service identity attributes, and ensures attribute keys match semantic conventions.
4. Storage indexes traces for trace_id lookup, metrics for aggregation by dimensions, and logs for field filtering and trace correlation.
5. A query finds elevated error rate in metrics, then retrieves traces in the same time window for the same service identity, and finally pulls correlated logs using trace_id.

The lifecycle is systematic because each stage enforces a contract: meaning is created in instrumentation, preserved in collection, made consistent in transformation, indexed in storage, and connected in query.

1.5 Common Failure Modes in Telemetry Pipelines and How to Detect Them

Telemetry pipelines fail in predictable ways: data disappears, data arrives but is unusable, or data arrives with the wrong meaning. The goal of this section is to help you spot those problems quickly by matching symptoms to likely causes, then using targeted checks to confirm.

Missing Telemetry After Instrumentation

A common failure mode is “nothing shows up,” even though the application is running. The usual culprits are exporter misconfiguration, sampling that removes everything, or network issues between the app and the collector.

Detection checklist

• Confirm the application exporter is enabled and points to the correct OTLP endpoint.
• Verify transport reachability from the application host to the collector.
• Check sampling settings at the SDK level and any collector-level sampling.
• Look for exporter errors in application logs.

Example:
If your service emits spans but you see zero traces in the backend, start with the collector: if the collector receives nothing, the problem is upstream. If the collector receives spans but the backend shows none, the problem is downstream (exporter, auth, or mapping).

Partial Data Due to Signal Split or Routing Rules

Pipelines often treat metrics, logs, and traces as separate flows. A routing rule that filters one signal can create a confusing “half working” system.

Detection checklist

• Verify each signal has its own pipeline and that receivers are attached to the correct one.
• Confirm processors that filter by attribute are not accidentally excluding a whole signal.
• Check that fan-out exporters are configured for every pipeline that needs them.

Example:
You may see metrics but not logs because a processor expects an attribute that logs don’t have yet. The fix is usually to enrich logs earlier in the pipeline or adjust the filter condition.

Semantic Drift from Inconsistent Attributes

Even when data arrives, it can be semantically wrong. Semantic drift happens when attribute names, units, or required fields vary across services.

Detection checklist

• Validate that resource attributes like service identity are consistent across deployments.
• Check that metric units and label keys match your conventions.
• Compare attribute presence rates across services to find outliers.

Example:
If one service reports http.status_code as a string and another as an integer, queries that assume numeric ordering will behave inconsistently. The collector can normalize types, but you must first detect the mismatch.

Broken Correlation from Missing Context Propagation

Distributed correlation fails when trace context is not propagated across boundaries. The symptom is traces that look like disconnected islands.

Detection checklist

• Inspect incoming requests for trace headers and confirm they are forwarded on outgoing calls.
• Check that messaging systems carry trace context in message metadata.
• Verify that span links or parent-child relationships are formed as expected.

Example:
A request enters Service A and Service B creates a new trace instead of a child span. That typically means the outgoing call from A did not include the trace context headers, or the receiving side is not extracting them.

Backpressure, Batching, and Retry Loops

Performance-related failures can look like data loss. Batching delays can make it appear that telemetry is missing, while retry loops can cause duplicates or bursts.

Detection checklist

• Monitor collector queue sizes and dropped item counters.
• Check batch sizes and timeouts to understand ingestion latency.
• Look for repeated export failures that trigger retries.

Example:
If you see periodic spikes in exported spans and gaps between them, batching and retry timing may be creating a sawtooth pattern. Confirm by correlating collector logs with backend ingestion timestamps.

Time Skew and Out-of-Order Events

When clocks differ, you get confusing timelines. Spans may appear to start after they end, or logs may not align with traces.

Detection checklist

• Ensure hosts use synchronized time.
• Compare event timestamps across signals for the same request.
• Check for ingestion-time versus event-time confusion in your backend.

Example:
A log line that should match a span’s duration appears outside the span window. Often the issue is timestamp source differences rather than missing data.

Data Quality Issues from Over-Filtering or Over-Enrichment

Filters can remove too much, and enrichment can add incorrect fields. Both reduce trust in dashboards.

Detection checklist

• Track how many items each processor drops.
• Validate enrichment logic against a small set of known requests.
• Confirm that enrichment does not overwrite existing correct attributes.

Example:
A processor that sets service.name based on a header might overwrite the real service identity when the header is absent. The result is a “new service” in dashboards that shouldn’t exist.

A Practical Triage Flow

Use a short sequence that narrows the problem quickly.

1. Confirm receipt: Does the collector receive the signal at all?
2. Confirm processing: Do processors drop or transform it unexpectedly?
3. Confirm export: Do exporters succeed and authenticate?
4. Confirm meaning: Are semantic fields and correlation intact?

Example:
If the collector receives spans but correlation is broken, focus on context propagation and extraction rather than exporter credentials. If the collector receives nothing, focus on endpoint configuration, sampling, and network reachability.

Detection Example with a Minimal Diagnostic Output

When you need a fast sanity check, compare counts at each stage for a single service and time window.

Time window: 2026-03-25T10:00Z to 10:05Z
Service: checkout-api
Received at collector: traces=12,340 metrics=8,900 logs=0
After processors: traces=12,340 metrics=8,900 logs=0
Exported to backend: traces=12,330 metrics=8,880 logs=0
Backend visible: traces=12,330 metrics=8,880 logs=0

If logs are zero from “received at collector,” the issue is upstream instrumentation or OTLP export for logs. If logs are nonzero at the collector but zero in the backend, the issue is exporter configuration or backend ingestion mapping.

2.1 OpenTelemetry Components and Responsibilities

OpenTelemetry is a set of libraries and a data model that help you produce metrics, logs, and traces in a consistent way. The “components” are the moving parts that turn application events into structured telemetry, then move that telemetry through processing steps until it reaches a backend. Think of it as a pipeline with clearly named roles, so you can reason about where a field is added, transformed, or dropped.

The Core Roles in the OpenTelemetry Stack

1. Instrumentation libraries create telemetry from your code. They know how to measure time, count events, and capture context.
2. SDKs provide the runtime behavior: buffering, sampling, resource attribution, and the mechanics of creating spans and metric points.
3. APIs define what your code calls. APIs are stable entry points; SDKs are the implementation.
4. Context propagation carries trace identity across process boundaries so related work stays connected.
5. Collector receives telemetry over OTLP, runs processing, and exports to one or more destinations.
6. Exporters send telemetry out of the SDK or collector to a backend or another system.

A practical way to remember responsibilities: APIs ask for telemetry, SDKs produce telemetry, collectors refine telemetry, and exporters deliver telemetry.

How Data Flows End to End

At runtime, your application emits telemetry through the API. The SDK turns that into structured data using the data model rules. For traces, it creates spans and links them using the active context. For metrics, it records measurements and later aggregates them according to the SDK’s configuration. For logs, it attaches fields and can correlate them with trace context when you include identifiers.

When you use a collector, the SDK typically exports via OTLP to the collector. The collector then applies processors such as batching, attribute normalization, and filtering. Finally, exporters deliver the processed payload to the backend.

Example: Traces with Clear Ownership

In a typical service, you want one incoming request to produce a trace with multiple spans. The API call creates a span, the SDK manages its timing and lifecycle, and context propagation ensures child spans join the same trace.

Request arrives
  -> Extract trace context from headers
  -> Start server span (SDK)
  -> Call downstream service
     -> Inject trace context into outgoing headers
     -> Downstream starts client span (SDK)
  -> End spans
Export
  -> OTLP to collector
  -> Collector batches and exports

Example: Metrics and Responsibility Boundaries

Metrics often confuse people because “recording” and “exporting” are separate steps. Your code records measurements (API + instrumentation). The SDK decides how to aggregate them (for example, sum and count over time windows). The exporter or collector then ships aggregated points.

A common best practice is to keep metric labels consistent with semantic conventions. If you vary label keys or formats across services, the backend will treat them as different dimensions, and your dashboards will quietly become less useful.

Example: Logs and Correlation Without Guesswork

Logs are not automatically correlated unless you provide the fields. A reliable pattern is to include trace identifiers in log records at the time you write them, using the active context.

When writing a log line
  - Include trace_id and span_id from active context
  - Include service.name and environment from resource
  - Keep message stable and structured fields separate

Responsibilities You Can Test

You can validate component behavior with small, deterministic checks:

• API/SDK boundary: confirm spans start and end when expected.
• Context propagation: confirm a downstream request receives the same trace identity.
• Collector processing: confirm attribute normalization occurs before export.
• Exporter delivery: confirm payloads arrive and are not dropped due to auth or endpoint mismatch.

A good mental model is that each component has a narrow job. When something is missing in the backend, you can usually trace the problem to one responsibility boundary rather than chasing ghosts across the whole system.

2.2 SDKs Exporters Receivers Processors and Pipelines

OpenTelemetry’s pipeline is easiest to understand as a set of roles that pass telemetry through a sequence of stages. The SDKs create telemetry, the Collector receives it, processors transform it, and exporters deliver it. The “pipeline” is the wiring that connects these roles.

SDKs Create Telemetry with Context

SDKs live in your application or runtime. They produce spans, metrics, and logs, and they attach two kinds of metadata that matter later in the Collector: resource attributes (who produced it) and instrumentation scope (what library produced it).

A span is more than a timer. It carries a trace identity, a span identity, a span kind, and attributes describing the operation. When you create a span inside a request handler, you also decide how it relates to the incoming request context. That relationship is what makes distributed correlation work.

Metrics and logs follow the same principle: the SDK emits structured data with consistent keys, not free-form strings. If you keep attribute names stable at the SDK boundary, the Collector can focus on routing and normalization rather than guessing.

Receivers Accept Data and Standardize Entry

In the Collector, receivers are the intake points. The most common setup uses OTLP, where the Collector listens for gRPC or HTTP requests. A receiver’s job is to accept incoming telemetry and convert it into the Collector’s internal format.

Operationally, receivers define where data arrives and what it looks like when it enters the pipeline. If you misconfigure endpoints or protocols, you’ll see “nothing arrived” rather than “arrived but wrong,” which is why receiver configuration is the first place to check.

Processors Transform Data in Ordered Chains

Processors run after reception and before export. They are ordered, so the sequence matters. A typical chain might enrich resource attributes first, then filter noisy data, then batch for efficiency.

Common processor categories:

• Enrichment: add or derive attributes, such as service identity or environment tags.
• Normalization: rename keys, standardize units, or align attribute formats.
• Filtering: drop spans or logs that match rules, reducing cost and clutter.
• Batching and Retry: improve throughput and resilience without changing meaning.

A practical example is attribute normalization. Suppose one service uses http.status_code and another uses status_code. If you normalize early in the pipeline, downstream queries become consistent.

Exporters Deliver Telemetry to Backends

Exporters send telemetry out of the Collector. They handle backend-specific protocol details, authentication, and any required mapping to the backend’s expectations.

An exporter should not be treated as a second chance to fix broken semantics. If you rely on the exporter to “make it work,” you’ll end up with silent inconsistencies. Instead, use processors to ensure the data matches the semantic conventions you intend to follow.

Pipelines Wire Everything Together by Signal

Pipelines define which receivers feed which processor chain and which exporters, typically per signal type. Metrics, logs, and traces usually have separate pipelines because their processing needs differ.

Here is a compact example showing the wiring concept:

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch: {}
  attributes:
    actions:
      - key: service.name
        action: update
        value: my-service
exporters:
  otlphttp:
    endpoint: https://backend.example/v1/otlp
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes, batch]
      exporters: [otlphttp]

The ordering is explicit: attributes runs before batch. That matters because batching groups items; you want the final attributes included in the batched payload.

Example: One Request, Three Signals, One Correlation Story

Consider an HTTP request handled by your service. The SDK creates a span for the handler, emits metrics for request duration, and records a log event for a notable outcome. Each signal includes resource attributes and shares the trace identity where applicable.

When the Collector receives the data, the traces pipeline can enrich or filter spans, while the metrics and logs pipelines can apply their own normalization rules. The key is that the pipeline design keeps signal-specific logic separate while preserving correlation fields.

Case Study: Why Processor Order Prevents Confusing Results

If you batch before you normalize attributes, you might still “see data,” but queries can show mixed attribute values across batches. By normalizing first, you ensure every exported item already follows the same key scheme. It’s a small ordering choice that prevents a lot of head-scratching later.

2.3 Context Propagation Concepts and Span Relationships

Context propagation is how trace context travels with work as it moves through your system. In OpenTelemetry, that context typically includes a trace identifier and the “current span” identifier, plus flags that affect sampling. The key idea is simple: when code starts new work, it should either create a new span as a child of the current span or continue an existing span when that’s the right model.

What “Context” Means in Practice

A context is a bundle of values associated with the currently executing operation. In OpenTelemetry, the most important values are:

• Trace identifiers: trace id and span id that let you correlate related telemetry.
• Span linkage: the parent-child relationship that determines where the new span sits in the trace graph.
• Sampling decision: whether the trace is recorded, which affects whether spans are created or dropped.

In code, you usually don’t pass these values manually. Instead, the SDK keeps a “current context” that is set when a span becomes active, then read when you create child spans.

Span Relationships You Actually Need

OpenTelemetry uses span relationships to represent how work is structured.

• Parent-child: the common case for synchronous request handling. A child span represents work performed within the parent span’s lifetime.
• Span kind: indicates the role of the span (client, server, producer, consumer, internal). Span kind helps interpret directionality in the trace.
• Links: used when you cannot express a strict parent-child relationship, such as when consuming a message that was produced elsewhere without a direct call stack.

Parent-child relationships are great for “call graph” style flows. Links are great for “correlation without causality” flows, like batch processing where the consumer handles many messages.

The Life Cycle of a Trace Context

A trace context typically enters your system at a boundary, then moves outward.

1. Incoming boundary: an HTTP request arrives with headers carrying trace context.
2. Server span creation: the server extracts the context and starts a server span as the active span.
3. Downstream calls: when the server calls another service, it injects the current context into outgoing headers.
4. Continuation: the downstream service extracts the headers and starts its own server span as a child of the upstream span.

If you skip extraction or injection at a boundary, you’ll get broken traces: spans still exist, but the graph won’t connect.

Example: HTTP Request with Correct Parent-Child Structure

Imagine Service A receives an HTTP request and calls Service B.

• Service A starts a server span for the incoming request.
• Service A creates a client span for the HTTP call to Service B.
• Service A injects the trace context into the outgoing HTTP headers.
• Service B extracts the headers and starts its own server span.

Result: Service B’s server span becomes a child of Service A’s client span, which is itself a child of Service A’s original server span. That gives you a trace graph that matches the real call chain.

Example: Async Messaging with Links Instead of Parent-Child

Now consider a queue. Service A publishes a message, and Service C later consumes it.

• Publishing creates a producer span and injects trace context into message metadata.
• Consumption extracts context and can create a consumer span.

If the consumer can treat the extracted context as the “current” parent, you get parent-child. If the consumer processes multiple messages together or the metadata is insufficient to define a single parent, you use links to associate the consumer span with one or more producing spans.

Common Mistakes and Their Symptoms

• Creating spans without the active context: child spans appear as separate traces or attach to the wrong parent.
• Extracting but not injecting: downstream services start new traces because they never receive the upstream context.
• Injecting but not extracting: upstream spans exist, but the trace graph stops at the boundary.
• Using parent-child where links fit better: you force a causality model that doesn’t match the system’s structure, which makes analysis misleading.

A Practical Mental Model

Treat propagation as “carrying the current span identity across boundaries.” When you can represent a call stack, use parent-child. When you can only represent correlation, use links. Span kind tells you which direction the work flows, while relationships tell you how the trace graph should be connected.

2.4 Resource Attributes and Instrumentation Scope Semantics

Resource attributes and instrumentation scope semantics are the two knobs that decide how telemetry is grouped, attributed, and interpreted. If you get them right, dashboards become consistent and debugging becomes less of a scavenger hunt. If you get them wrong, everything still arrives, but it arrives in the wrong buckets.

Resource Attributes as the “Where” and “Who”

A resource describes the entity that owns the telemetry. In practice, that usually means the process, service, or deployment environment that produced the data. Resource attributes are attached to telemetry at export time and are intended to be stable for the lifetime of the emitting process.

Common resource attributes include:

• service.name: The logical service identifier used for grouping.
• service.namespace and service.instance.id: Optional identifiers that help disambiguate multi-tenant or multi-instance setups.
• deployment.environment: Values like production, staging, or test.
• cloud.*, k8s.*, host.*: Environment-specific metadata such as region, cluster, or node.

A practical rule: use resource attributes for identity and topology, not for per-request details. Per-request details belong on spans, events, metrics data points, or log records.

Example: Stable Identity vs Per-Request Variability

If you attach http.route as a resource attribute, it will explode cardinality because it changes frequently. Instead, put http.route on spans (or on metrics labels if you truly need it), and keep resource attributes focused on stable identity like service.name and deployment.environment.

Instrumentation Scope as the “Which Library”

Instrumentation scope identifies the component that created the telemetry within the process. It is the semantic home for the instrumentation library name and version, and it helps separate telemetry produced by different SDKs or custom instrumentation.

Instrumentation scope is especially useful when:

• You have multiple instrumentation libraries in one service.
• You want to distinguish auto-instrumentation from manual instrumentation.
• You need to track changes when upgrading an instrumentation library.

A useful mental model: resource attributes answer “which service and environment,” while instrumentation scope answers “which instrumentation produced this.”

How Grouping Works in Practice

Backends typically group data using a combination of resource attributes and instrumentation scope. That means your choices affect:

• Which series appear together for metrics.
• How traces are filtered by service.
• How logs are correlated and queried.

To keep grouping predictable, treat resource attributes as your primary partition key and instrumentation scope as your secondary partition key.

Example: Consistent Resource and Scope for a Web Service

Imagine a web service running in Kubernetes. You want every trace and metric to be grouped under the same service identity, while still distinguishing the HTTP auto-instrumentation from a custom database span wrapper.

• Resource attributes:

  • service.name = "checkout-api"
  • deployment.environment = "staging"
  • k8s.cluster.name = "cluster-a"
  • k8s.namespace.name = "payments"

• Instrumentation scope:

  • HTTP auto-instrumentation scope: name = "opentelemetry-instrumentation-http", version = "x.y.z"
  • Custom DB wrapper scope: name = "checkout-db-wrapper", version = "1.3.0"

Now a query for service.name=checkout-api returns a coherent view, and a breakdown by instrumentation scope helps you verify which part of the system produced a particular span pattern.

Example: Avoiding Cardinality Traps

A common mistake is to place request-specific values into resource attributes. For instance, deployment.environment is stable, but user.id is not. If you put user.id into resource attributes, you create a new resource identity for every user, which makes aggregation and storage expensive.

Instead:

• Put user.id on spans as an attribute only if you truly need it for debugging.
• Prefer a stable grouping key like user.segment if you need analytics.
• For metrics, use aggregated dimensions that you can tolerate at scale.

Operational Semantics That Matter

When you standardize resource attributes across services, you get consistent service-level dashboards without per-service exceptions. When you standardize instrumentation scope naming and versioning, you can interpret changes in telemetry patterns without guessing which library produced them.

In short: resource attributes define the stable identity of the telemetry producer, and instrumentation scope defines the stable identity of the instrumentation itself. Together, they make vendor-neutral data behave like a well-labeled dataset instead of a pile of events.

2.5 Collector Deployment Patterns for Local and Centralized Processing

Collector placement decides where you pay for CPU, where you enforce consistency, and how quickly you can react to bad telemetry. A good rule: keep instrumentation simple, then use the collector to normalize, filter, and route data.

Local Processing Pattern

In a local pattern, each host or cluster runs an OpenTelemetry Collector near the sources. The collector receives OTLP from instrumented services, performs lightweight processing, and exports to one or more backends.

When it fits

• You want to reduce cross-network traffic by filtering or sampling early.
• You need per-environment resource enrichment close to where service identity is known.
• You want faster feedback loops when a single service misbehaves.

What to do locally

• Add or normalize resource attributes such as service.name, service.namespace, and deployment.environment.
• Apply basic filtering to drop noisy metrics or logs that you know you will never query.
• Batch and retry to smooth short bursts.

What to avoid locally

• Heavy transformations that require large lookups or complex aggregation rules.
• Central policy enforcement that must be identical across many environments.

Example local pipeline idea

• Receivers: OTLP over gRPC
• Processors: batch, resource detection, attribute normalization
• Exporters: one metrics backend and one trace backend

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  batch: {}
  resource: {attributes: []}
exporters:
  otlphttp:
    endpoint: https://backend.example/otlp
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Centralized Processing Pattern

In a centralized pattern, instrumented services send OTLP to a shared collector tier. That tier applies consistent processing rules and routes to backends.

When it fits

• You need uniform semantic normalization across many teams.
• You want one place to enforce attribute requirements and naming conventions.
• You prefer simpler service-side configuration.

What to do centrally

• Enforce attribute presence and consistent naming for metrics, spans, and logs.
• Perform transformations that require shared context, such as mapping legacy attributes into semantic conventions.
• Route by tenant, environment, or service group.

What to avoid centrally

• Per-host resource detection that depends on local filesystem or network metadata, unless you can guarantee the same inputs.
• Extremely chatty enrichment that increases payload size before export.

Example centralized pipeline idea

• Receivers: OTLP over HTTP
• Processors: batch, attributes normalization, filtering, routing
• Exporters: multiple backends by signal type

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch: {}
  attributes:
    actions:
      - key: service.namespace
        action: upsert
        value: "payments"
exporters:
  logging: {verbosity: detailed}
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [logging]

Hybrid Pattern with Two Tiers

A hybrid pattern combines both: local collectors handle buffering and lightweight normalization, while centralized collectors enforce cross-service consistency.

Why it works

• Local collectors reduce the blast radius of network issues by buffering and retrying.
• Central collectors become the policy gate, so dashboards and alerts stay consistent.

Practical division of labor

• Local: batch, basic filtering, minimal resource enrichment.
• Central: semantic normalization, attribute mapping, routing, and final export.

Operational Checks That Prevent “It Looks Fine” Failures

1. Verify pipeline separation by signal: traces, metrics, and logs should not share processors that assume a specific data shape.
2. Confirm resource attribute consistency: check that service.name and deployment.environment appear on every exported record.
3. Measure queue behavior: if batching hides drops, you will only notice when dashboards go quiet.
4. Test routing rules with sample payloads: a single mis-typed attribute can send all logs to the wrong backend.

Case Study: Choosing a Pattern for a Small Platform

A team runs 12 services across two environments. They want consistent service naming and want to avoid configuring every service with complex routing rules.

• They deploy a local collector per node to handle OTLP ingestion, batch, and basic resource detection.
• They deploy a centralized collector tier to enforce attribute mapping into semantic conventions and to route metrics and traces to different backends.
• They keep local collectors lean so CPU spikes from enrichment do not affect application latency.

The result is predictable dashboards because the centralized tier guarantees consistent attributes, while the local tier keeps ingestion resilient during short network hiccups.

3.1 OTLP Protocol Overview for Metrics Logs and Traces

OTLP, short for OpenTelemetry Protocol, is the standard way OpenTelemetry components send telemetry data to a Collector or directly to a backend. It matters because it keeps instrumentation vendor-neutral while still letting you choose transports, security, and processing rules.

What OTLP Sends and How It Is Organized

OTLP carries three signal types: metrics, logs, and traces. Each signal has its own payload shape, but the overall delivery pattern is consistent: the sender batches records, attaches metadata, and transmits them to an OTLP endpoint.

A practical way to think about OTLP is “records in, batches out.” Your application produces individual spans, metric points, and log records. The SDK and/or Collector groups them into batches to reduce overhead and improve throughput.

Transport Options and Their Operational Impact

OTLP commonly uses two transports: gRPC and HTTP.

• gRPC is efficient for high-volume streaming-style workloads. It also supports strong typing for request/response semantics.
• HTTP is often simpler to route through existing HTTP infrastructure and load balancers.

In both cases, you configure an endpoint and credentials (if needed). The payload content is the same conceptually, but the wire format and request patterns differ.

Endpoint Shape and Request Routing

An OTLP endpoint is typically the host and port where the Collector listens. The Collector exposes separate handlers for traces, metrics, and logs, even if you use a single base address.

A common operational best practice is to keep the endpoint stable and route by signal type at the Collector. That way, you can change processing pipelines without redeploying applications.

Payload Structure at a Glance

OTLP requests include:

• Resource information: who produced the telemetry (service name, environment, host identity).
• Instrumentation scope: which library or component created the data.
• Signal-specific records:
  • Traces: spans with trace and span identifiers, timing, attributes, events, and status.
  • Metrics: metric descriptors plus datapoints with timestamps and values.
  • Logs: log records with timestamps, severity, body, and attributes.

This separation is what makes semantic conventions work across signals. Resource attributes provide consistent identity, while signal records carry the details you query.

Batching, Timing, and Backpressure

OTLP senders batch data to reduce per-request overhead. Batching introduces two practical concerns:

1. Latency: larger batches can delay visibility.
2. Backpressure: if the Collector is slow, buffers fill.

A good rule of thumb is to tune batch size and export interval based on your tolerance for delay and your tolerance for memory growth. The Collector’s queueing and retry behavior also affects end-to-end delivery.

Error Handling and Delivery Semantics

OTLP export is not “fire and forget.” When the Collector rejects a request (for example, due to malformed data or auth failures), the sender receives an error and can retry depending on configuration.

For traces and metrics, partial acceptance can happen at the batch level. That means you should validate that your Collector logs show consistent ingestion rather than only checking that “something arrived.”

Example: Minimal OTLP Export Setup Conceptually

Imagine an application exporting all three signals to a Collector. The application SDK produces telemetry, batches it, and sends OTLP requests to the Collector endpoint.

On the Collector side, the OTLP receiver accepts requests, then forwards them into signal-specific pipelines. Those pipelines can normalize attributes, filter noise, and export to your storage.

Even without showing configuration syntax, the key idea is consistent: OTLP is the intake contract; pipelines are the transformation and delivery logic.

Example: Traces Request Content You Should Expect

For a single incoming request in your application, OTLP traces typically include:

• A span representing the server-side handler.
• A span representing downstream work (client calls, database operations, or messaging).
• Shared trace identifiers so the spans correlate in queries.
• Attributes that match semantic conventions, such as HTTP method and route, when applicable.

If you see spans arriving without expected correlation identifiers or missing key attributes, the issue is usually in instrumentation or semantic mapping, not in OTLP itself.

Example: Metrics and Logs Share Resource Identity

Metrics datapoints and log records both reference the same resource identity fields. That consistency is what lets you correlate “what happened” (logs), “how it behaved” (metrics), and “where it happened” (traces) using the same service and environment attributes.

When you design your resource attributes carefully, OTLP becomes a reliable transport layer rather than a source of inconsistency.

3.2 gRPC Versus HTTP Transport Selection and Operational Tradeoffs

Choosing between gRPC and HTTP for OTLP is mostly about how you want the network behavior to feel under load. Both can carry the same telemetry concepts, but they differ in framing, connection usage, error surfaces, and operational knobs.

Core Transport Differences That Matter

gRPC runs over HTTP/2 and uses a binary framing model. That typically means fewer bytes spent on headers and more predictable streaming behavior when many requests are in flight. HTTP/1.1 or HTTP/2 OTLP endpoints can also work, but the “shape” of requests and responses is different: you often see more request/response boundaries and different buffering behavior.

In practice, the biggest operational differences show up in four areas: connection management, payload framing, error handling, and middlebox behavior.

Connection Management and Concurrency

With gRPC, HTTP/2 multiplexing lets one connection carry many concurrent RPCs. That reduces connection churn and can smooth out bursts when your application exports frequently. The tradeoff is that you need to ensure your load balancer and network path handle HTTP/2 correctly; otherwise you may see intermittent failures that look like “random” export drops.

With HTTP OTLP, you may end up with more frequent new requests, depending on client settings and server behavior. This can be perfectly fine, especially when your exporter batches and sends less often. The operational question becomes: do you prefer fewer long-lived connections (HTTP) or fewer connection setups with multiplexing (gRPC)?

Payload Framing and Backpressure

gRPC’s framing can make it easier for the transport to handle partial progress and streaming patterns. Even when you are not streaming telemetry, the underlying model tends to behave well when the client is producing data continuously.

HTTP payloads are typically sent as discrete bodies. If your exporter batches too aggressively, you can create large request bodies that increase latency and memory pressure. If you batch too conservatively, you increase request count and overhead. Either way, the transport amplifies the consequences of poor batching.

A practical rule: tune batching first, then choose the transport. Transport choice changes the “cost curve,” but batching determines where you sit on it.

Error Handling and Retry Semantics

gRPC surfaces failures via gRPC status codes and often includes richer details about why an RPC failed. HTTP surfaces failures via HTTP status codes and may include different error bodies depending on the server.

Operationally, you want your exporter to treat transient network issues differently from permanent configuration problems. For example:

• If you see timeouts or connection resets, retries can help, but you must cap retry attempts to avoid amplifying load.
• If you see authentication errors or schema/validation failures, retries usually waste time and can flood logs.

Because error surfaces differ, you should verify how your collector reports them. A collector that logs “exporter failed” without the underlying status code makes both transports harder to troubleshoot.

Middlebox Compatibility and Timeouts

Some environments are less comfortable with HTTP/2. Common culprits include older proxies, strict idle timeouts, and load balancers that mishandle multiplexed traffic. When that happens, gRPC can fail in ways that are consistent but non-obvious, such as repeated reconnects.

HTTP often has broader baseline compatibility, especially when HTTP/1.1 is used. The tradeoff is that you may pay more overhead per request and see more sensitivity to client-side timeouts.

If you operate through a corporate proxy, test both transports in the same network path. Don’t compare them in a lab that bypasses the production proxy.

Example: Choosing Based on Export Pattern

Suppose you have a service that exports every 5 seconds with moderate batch sizes.

• If you expect many concurrent requests and want stable connection behavior, gRPC is usually a strong fit because multiplexing reduces overhead during bursts.
• If your environment has strict HTTP/2 constraints or you see frequent reconnects, HTTP can be simpler to operate, especially if batching keeps request sizes reasonable.

Example: Operational Checklist

• Confirm the collector endpoint supports the chosen OTLP transport.
• Verify network path supports HTTP/2 end-to-end if using gRPC.
• Set exporter timeouts to values that match your network latency budget.
• Ensure batching is configured so request bodies stay within safe limits.
• Check collector logs for the actual failure status, not just a generic “failed to export.”

Practical Decision Summary

Pick gRPC when you want efficient multiplexing and you trust the network path to handle HTTP/2 reliably. Pick HTTP when you need maximum compatibility or when your environment’s HTTP/2 handling is uncertain. In both cases, correct batching and clear failure reporting matter more than the transport label.

3.3 Endpoint Configuration and Network Considerations

Endpoint configuration is where “it works on my machine” meets the real world. OTLP endpoints define where telemetry goes, how it’s transported, and what the network will allow. A good setup makes failures obvious, keeps latency predictable, and avoids silent data loss.

Transport Choice and Endpoint Shape

OTLP commonly uses either gRPC or HTTP. The collector and SDKs both accept OTLP endpoints, but the exact URL or host/port fields differ by transport.

• gRPC typically uses a host and port, with the client speaking HTTP/2 under the hood.
• HTTP typically uses a full URL path to the OTLP receiver.

A practical rule: treat the endpoint as a contract. If you change transport, you often change configuration keys, TLS settings, and sometimes proxy behavior.

Host, Port, and Path Details That Matter

For gRPC, you usually specify something like host:port. For HTTP, you specify a URL including the receiver path. If you’re behind a reverse proxy, the path must match what the proxy forwards.

Common pitfalls:

• Wrong port: the collector is listening on one port, but the SDK exports to another.
• Wrong path: HTTP OTLP receivers expect a specific route; a missing or extra segment breaks delivery.
• IPv4 versus IPv6 mismatch: some environments resolve localhost differently than expected.

TLS, Certificates, and Verification Behavior

TLS is not just “turn it on.” You need to decide whether the client verifies the server certificate.

• In production, prefer certificate verification so misrouting doesn’t quietly succeed.
• If you use internal CAs, ensure the client trusts them.
• For local testing, it’s tempting to disable verification; do it only in controlled environments and keep the setting explicit.

When debugging, confirm three things: the scheme (http vs https), the certificate chain, and the hostname used for verification.

Proxies, Load Balancers, and Connection Lifetimes

Network middleboxes can change behavior in ways that look like application bugs.

• HTTP proxies may require explicit configuration for CONNECT or may not support HTTP/2 end-to-end.
• Load balancers can close idle connections; if the exporter keeps a long-lived connection, you may see intermittent failures.
• NAT gateways can run out of ephemeral ports under high telemetry volume.

A simple operational check is to compare exporter error logs with collector receiver logs. If the collector never receives anything, the issue is upstream networking or endpoint addressing.

Timeouts, Retries, and Backpressure

Telemetry should not block the application indefinitely. Exporters typically buffer and retry, but the buffering strategy has limits.

Key knobs:

• Timeout: how long the exporter waits for a response.
• Retry policy: how it behaves after transient failures.
• Queue size: how much data can accumulate before dropping.

A useful mental model: timeouts and retries trade off between delivery and resource usage. If timeouts are too short, you’ll retry aggressively. If they’re too long, you’ll hold resources while the network is unhealthy.

Example Collector Endpoint Configuration

Below is a minimal collector receiver setup for OTLP over both transports. Adjust ports and TLS to match your environment.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

If you place the collector behind a load balancer, ensure the LB forwards both ports (or only the one you use) and preserves the protocol.

Example SDK Exporter Endpoint Configuration

This example shows OTLP over HTTP with a full URL. The exact path must match the collector’s OTLP HTTP receiver.

exporters:
  otlphttp:
    endpoint: http://collector.example.internal:4318
    headers:
      x-tenant-id: "acme"

If you switch to HTTPS, update the scheme and ensure certificate trust is configured.

Practical Debugging Checklist

When telemetry disappears, avoid guessing. Verify reachability first, then protocol correctness, then TLS, then timeouts.

1. Confirm the collector is listening on the expected port(s).
2. Confirm the SDK endpoint matches transport and path.
3. If TLS is enabled, confirm the certificate chain and hostname.
4. Check exporter logs for timeout or retry patterns.
5. Check collector receiver logs for incoming requests and decoding errors.

A well-configured endpoint turns “mysterious missing data” into a short list of concrete, testable causes.

3.4 Authentication and Secure Transport for OTLP Endpoints

OTLP endpoints are where telemetry leaves your environment, so treat them like a network boundary: authenticate who is allowed to send data, and protect the data in transit. In practice, you’ll combine transport security (usually TLS) with application-level authentication (often tokens or mTLS). The goal is simple: only authorized clients can connect, and intermediaries can’t read or tamper with the payload.

Core Threats and What Each Control Prevents

Start with a quick mapping from problem to control:

• Eavesdropping: Without TLS, anyone on the path can read telemetry payloads. TLS provides confidentiality.
• Impersonation: Without authentication, any host can send fake telemetry. Authentication blocks unauthorized clients.
• Tampering: Without integrity protection, payloads can be modified in transit. TLS integrity prevents this.
• Replay: Some attackers can resend captured traffic. Strong TLS plus short-lived credentials reduces risk.

Secure Transport with TLS

Most OTLP deployments use HTTPS (HTTP/2) or gRPC over TLS. Configure the OTLP exporter to use TLS and validate the server identity.

Key choices:

• Server certificate validation: Prefer system trust stores or explicitly pinned CA certificates. Disabling verification is a common “it works on my machine” trap.
• Hostname matching: Ensure the endpoint hostname matches the certificate subject or SAN.
• Cipher and protocol defaults: Use library defaults unless you have a documented reason to change them.

Example: configuring an OTLP exporter to use TLS with a custom CA.

exporters:
  otlp/secure:
    endpoint: "otel-collector.example.com:4317"
    tls:
      ca_file: "/etc/ssl/certs/ca-bundle.crt"
      insecure: false

If you’re using HTTP OTLP (typically port 4318), the same principles apply: HTTPS with certificate validation.

Authentication Options for OTLP Endpoints

Authentication answers: “Who is allowed to send telemetry?” Common approaches include:

1. Bearer tokens: A shared token or per-client token placed in an Authorization header.
2. API keys: Similar to bearer tokens but often managed as keys with rotation policies.
3. mTLS client certificates: Each client has its own certificate; the server validates it.

Bearer tokens are operationally straightforward, while mTLS is stronger for environments where you want identity tied to certificates rather than secrets.

Bearer Token Authentication with OTLP

When using bearer tokens, ensure the token is stored securely (for example, mounted as a file or injected via a secret manager) and never hard-coded in config files committed to source control.

Example: adding a bearer token to an OTLP exporter.

exporters:
  otlp/secure:
    endpoint: "otel-collector.example.com:4317"
    headers:
      Authorization: "Bearer ${OTLP_TOKEN}"

On the collector side, you must configure an authentication mechanism that checks the Authorization header. The exact configuration depends on the collector’s authentication capabilities and your deployment pattern, but the principle is consistent: reject requests without a valid token before accepting payloads.

mTLS Authentication with Client Certificates

With mTLS, the server verifies the client certificate, and the client verifies the server certificate. This gives you mutual identity and reduces the chance of “token leaked, now anyone can send.”

Example: enabling client certificate authentication from the exporter.

exporters:
  otlp/secure:
    endpoint: "otel-collector.example.com:4317"
    tls:
      ca_file: "/etc/ssl/certs/ca-bundle.crt"
      cert_file: "/etc/otel/client.crt"
      key_file: "/etc/otel/client.key"
      insecure: false

On the collector side, you configure it to require client certificates and validate them against a trusted CA. Also ensure certificate rotation is planned so telemetry doesn’t silently stop when credentials expire.

Authorization Boundaries and Least Privilege

Authentication proves identity; authorization decides what that identity can do. For telemetry, least privilege usually means:

• Allow only the expected services or namespaces to send to a given collector endpoint.
• Restrict access to administrative endpoints separately from OTLP ingestion.
• Keep different environments (dev, staging, prod) on separate credentials and endpoints.

A practical rule: if you can’t explain which service owns a credential, you probably can’t operate it safely.

Practical Validation Checklist

Before you trust the pipeline, verify behavior under both success and failure:

• Success path: confirm the exporter connects and data arrives with the expected resource attributes.
• Missing credentials: confirm requests without Authorization or client cert are rejected.
• Wrong credentials: confirm invalid tokens or untrusted client certs are rejected.
• Certificate issues: confirm expired or mismatched certificates fail fast rather than falling back to insecure modes.

A secure OTLP endpoint should fail predictably. When it doesn’t, you’ll usually find either certificate verification disabled, credentials injected incorrectly, or the collector not enforcing the expected auth check.

3.5 Validating OTLP Payloads with Practical Debugging Techniques

Validating OTLP payloads is less about “it sends” and more about “it means what you think it means.” The goal is to confirm three things: the exporter is producing valid OTLP, the collector is accepting it, and the resulting telemetry matches the semantic intent (names, attributes, and relationships).

What to Validate First

Start with the smallest, most observable checkpoints.

1. Transport success: the collector receives something at the configured OTLP endpoint.
2. Schema validity: the payload is well-formed OTLP for the signal type (metrics, logs, traces).
3. Semantic correctness: key fields exist and follow expected naming patterns.
4. Correlation correctness: trace and span identifiers line up so downstream queries can join data.

A common mistake is to validate only step 1. You can get “successful export” while still producing empty resource attributes, missing required fields, or mismatched trace context.

A Systematic Debugging Workflow

Step 1: Confirm Signal Routing

Make sure the exporter is targeting the right endpoint and protocol.

• If you use OTLP/gRPC, verify the port and that the collector receiver is configured for gRPC.
• If you use OTLP/HTTP, verify the path and that the receiver is configured for HTTP.

A quick sanity check is to temporarily enable verbose logging on the collector receiver and watch for “received” events tied to the correct signal type.

Step 2: Inspect Payload Shape Without Guessing

When you can’t easily view the raw OTLP, you still can validate shape by checking what the collector forwards.

• Enable debug-level logs for the pipeline components.
• Use a “debug exporter” or a local logging exporter to print processed telemetry.

You’re looking for structural markers:

• Traces: resource attributes, scope/instrumentation scope, spans with span kind, trace id, span id, parent span id when applicable.
• Metrics: resource attributes, metric name, instrument type, data points with timestamps and values.
• Logs: resource attributes, log body, severity, trace id and span id when correlation is expected.

Step 3: Validate Required Fields and Naming Consistency

Semantic conventions are strict enough that small deviations can break dashboards.
Check these categories:

• Resource identity: service name, service namespace (if used), service version, deployment environment.
• Attribute naming: consistent prefixes and stable keys for dimensions.
• Units and types: metric values should be numeric with correct unit expectations.

If you see missing resource attributes, it often means resource detection didn’t run or the exporter didn’t set them.

Step 4: Validate Trace Context and Parentage

For traces, confirm that:

• trace id stays constant across spans in the same request flow.
• parent-child relationships match the expected call graph.
• span kind aligns with client/server roles.

For logs, confirm that trace id and span id are present when your logging instrumentation is configured to correlate.

Practical Examples

Example: Detecting a Wrong Endpoint

If your collector receiver is configured for OTLP/HTTP but the exporter is sending OTLP/gRPC, you may still see “export attempts” but no usable telemetry.

• Symptom: debug output shows no spans/metrics/logs.
• Fix: align exporter protocol with collector receiver configuration.

Example: Spotting Missing Resource Attributes

Suppose your backend expects service.name but the debug output shows only generic host attributes.

• Symptom: dashboards show “unknown service” or empty groupings.
• Fix: ensure resource detection is enabled and that the exporter/SDK sets service identity.

Example: Catching Broken Parentage

If you see spans with the same trace id but no parent-child links, it usually means context propagation failed.

• Symptom: traces look like a flat list.
• Fix: verify propagation headers are injected on outgoing requests and extracted on incoming requests.

Minimal Collector Debug Configuration

Use a debug exporter to print what the collector actually forwards.

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      exporters: [debug]
    logs:
      receivers: [otlp]
      exporters: [debug]

This configuration helps you validate structure and correlation at the collector boundary, which is where many “it exported but nothing makes sense” issues become obvious.

What “Good” Looks Like

Good validation results in a consistent story:

• The collector receives the intended signal type.
• The payload contains the expected resource identity and instrumentation scope.
• Traces show correct trace id and parentage.
• Metrics include the expected metric names and dimensions.
• Logs include trace correlation fields when configured.

Once those checks pass, you can trust that any remaining issues are usually backend mapping or query logic rather than OTLP payload correctness.

4.1 Semantic Conventions Principles and Attribute Naming Rules

Semantic conventions are the shared vocabulary that makes telemetry comparable across services, teams, and backends. The goal is simple: when you say “HTTP method” or “service name,” everyone means the same thing, and queries don’t turn into scavenger hunts.

Principle One: Use Meaning Before Convenience

Start with the concept you’re describing, then choose the attribute that represents it. If you invent a custom attribute like http_method_name, you may still be able to query it, but you’ve lost interoperability with dashboards, alerts, and tooling that expect the standard key.

Example:

• Prefer http.request.method = "GET" over request_method = "GET".
• Prefer db.system = "postgresql" over databaseType = "postgres".

This rule also affects value choices. For categorical fields, use the expected set of values (for example, standard HTTP methods), rather than free-form strings that differ by team.

Principle Two: Keep Keys Stable and Values Consistent

Attribute keys should not change meaning over time. If you rename service.name to something else, you break historical continuity and make correlation harder.

Example:

• Stable key: service.name always identifies the logical service.
• Stable value: service.name = "checkout-api" stays the same even if the deployment changes.

When you must change a value, do it by introducing a new attribute or by versioning at the source, not by silently repurposing the old key.

Principle Three: Model the Right Level of the System

Telemetry attributes often come from different “levels”: process, service, request, and operation. Semantic conventions help you place each attribute at the correct level so aggregation behaves predictably.

Example:

• Put identity at the resource level: service.name, service.namespace, service.instance.id.
• Put request details at the span level: http.target, http.status_code.
• Put database details at the span level: db.name, db.operation.

If you mix levels, you get confusing results like “average latency per instance” when you meant “per service.”

Principle Four: Prefer Standard Keys for Cross-Signal Correlation

Correlation works best when the same concept uses the same key across traces, metrics, and logs. Even if your backend can correlate by trace IDs, consistent attribute naming improves filtering and grouping.

Example:

• Use trace_id and span_id in logs only as needed for correlation, but keep the rest of the context aligned with the same semantic keys used in traces.

Principle Five: Use Attributes to Enable Query Patterns

A good attribute choice supports the queries you actually run. Think in terms of dimensions: what do you group by, filter by, and compare?

Example query patterns to design for:

• “Latency by route”: group by http.route and filter by service.name.
• “Errors by dependency”: group by db.system and db.operation.
• “Throughput by queue”: group by messaging.system and messaging.destination.

Attribute Naming Rules That Prevent Pain Later

Semantic conventions follow a few practical rules that you can apply even when you’re not sure which exact key to use.

1. Use the correct prefix family: keys are organized by concept families like http.*, db.*, messaging.*, service.*.
2. Use the expected casing and separators: keys are typically lowercase with dots as separators.
3. Choose the most specific attribute available: if you have http.route, don’t fall back to a generic http.path unless the standard expects otherwise.
4. Avoid mixing synonyms: don’t store the same idea under two keys (for example, both http.method and http.request.method) unless you have a clear reason.
5. Don’t overload a key: http.status_code should be an HTTP status code, not an application error code.

Example: From Raw Fields to Semantic Attributes

Suppose your application logs include method, path, and status. A semantic mapping turns that into query-friendly telemetry.

• Raw fields: method = "GET", path = "/v1/orders/123", status = 404
• Semantic attributes:
  • http.request.method = "GET"
  • http.target = "/v1/orders/123" (or http.route if you have a normalized route)
  • http.status_code = 404

Now a single query can group across services, because the keys mean the same thing everywhere.

Example: A Common Mistake and Its Fix

Mistake:

• http.status_code = "NOT_FOUND" (string instead of numeric code)

Fix:

• http.status_code = 404

This matters because numeric fields aggregate and filter differently than strings, and backends often apply type-aware processing.

Summary of the Practical Rules

Choose standard keys for standard concepts, keep keys and values stable, model attributes at the correct level, and design naming so your usual queries work without custom logic. If you do that, your telemetry becomes easier to interpret and harder to misread.

4.2 Defining Service Identity with Resource Attributes

Service identity is the part of your telemetry that answers one practical question: “Which service produced this data?” In OpenTelemetry, that answer is carried primarily by resource attributes. If you get this right, dashboards, alerts, and cross-signal correlation become straightforward. If you get it wrong, you’ll spend time reconciling “almost the same” services that differ by a single attribute.

What Resource Attributes Mean in Practice

A resource is a set of key-value attributes attached to telemetry emitted by an SDK. For example, when your application emits spans and metrics, the SDK attaches the same resource identity to both signals (unless you intentionally override it). This consistency matters because queries often group by service identity across metrics, logs, and traces.

A good resource identity is:

• Stable: it should not change per request.
• Descriptive: it should identify the service and its deployment context.
• Low cardinality: avoid per-user or per-request values.

The Minimum Set of Identity Attributes

Start with a small, reliable set. Commonly used attributes include:

• service.name: the human-readable service name.
• service.namespace: an optional grouping for organizations or platforms.
• service.instance.id: a stable identifier for a specific instance (pod, VM, or process).
• deployment.environment: values like production, staging, or development.

A practical rule: if you can’t explain the attribute to an on-call engineer in one sentence, it’s probably not the right attribute for resource identity.

Example Service Identity

Imagine an API service running in Kubernetes. You might set:

• service.name = orders-api
• service.namespace = ecommerce
• service.instance.id = orders-api-7f9c2a1b (pod name or a stable container id)
• deployment.environment = production

Now every span, metric, and log record emitted by that process carries the same identity.

Choosing service.name Without Regret

service.name is the anchor for most grouping. Keep it consistent across languages and deployments. If you rename it, you effectively create a new service in your observability backend.

A simple convention works well:

• Use a short, lowercase, hyphenated name like orders-api.
• Keep it aligned with how your team refers to the service.
• Avoid including environment in service.name if you already have deployment.environment.

Using service.namespace for Organizational Grouping

service.namespace helps when you have multiple teams or platforms. For example, you might use ecommerce for the business domain and keep service.name focused on the specific service.

If you don’t have a natural namespace, it’s acceptable to omit it. The key is avoiding a “fake namespace” that forces you to maintain a taxonomy later.

When service.instance.id Matters

service.instance.id is useful when you need to answer questions like:

• Which instance is producing errors?
• Is a rollout causing a subset of pods to misbehave?

However, it can increase cardinality. Use a stable instance identifier, not a random value per process start unless you truly want that behavior.

In Kubernetes, pod names change during rescheduling. That’s okay if you treat instance identity as “this running unit right now.” If you need longer-lived identity, use a deployment-specific identifier plus a stable container id.

Advanced Details That Prevent Subtle Bugs

Avoid Per-Request Values in Resource

Resource attributes are attached at the SDK level, so they should not vary per request. If you put request-specific data into resource attributes, you’ll create high-cardinality dimensions and make aggregation expensive.

Instead, use request-specific data as span attributes (for traces) or log fields (for logs). Resource identity should remain the same while the request flows.

Keep Identity Consistent Across Signals

If your metrics and traces are emitted by different SDK instances or different processes, ensure they share the same resource identity configuration. A common failure mode is setting service.name in one place but forgetting it in another, leading to split dashboards.

Example: Resource Identity Configuration Pattern

Below is a conceptual pattern for setting resource attributes. The exact API differs by language, but the idea is consistent: create a resource once, reuse it, and attach it to the telemetry pipeline.

Resource attributes:
- service.name: orders-api
- service.namespace: ecommerce
- deployment.environment: production
- service.instance.id: orders-api-7f9c2a1b

Attach this resource to:
- tracer provider
- meter provider
- logger provider

A Quick Consistency Checklist

Before you ship, verify:

• service.name is identical across services that should be the same.
• deployment.environment is present and uses consistent values.
• service.instance.id is stable enough for your debugging needs.
• No request-specific values appear in resource attributes.

When these conditions hold, your observability system can group and correlate telemetry by service identity without surprises. That’s the boring part done correctly, which is exactly what you want.

4.3 Metrics Instrument Types and Aggregation Semantics

Metrics in OpenTelemetry are built from two ideas that work together: an instrument type tells you how values are recorded, and aggregation semantics tell you how those values are combined over time. If you get either wrong, dashboards will look plausible while telling the wrong story.

Instrument Types as Recording Behaviors

Think of an instrument as a contract between your code and the collector/backend.

• Counter records values that only increase. Use it for totals like requests received. The backend aggregates by summing increments.
• UpDownCounter can go up and down. Use it for gauges that represent a changing quantity but still want sum-based aggregation, such as in-flight jobs. The backend sums positive and negative deltas.
• Histogram records a distribution by bucketing observed values. Use it for latencies and payload sizes. The backend aggregates by counting observations per bucket and tracking totals.
• Gauge represents a value at a point in time. Use it for current queue depth or memory usage. The backend aggregates by choosing a strategy such as last value, min/max, or average depending on configuration.

A practical rule: if you can explain the metric as “this never decreases,” start with a Counter. If you can explain it as “this is a current reading,” start with a Gauge. If you can explain it as “this is a sample of a distribution,” start with a Histogram.

Aggregation Semantics as Time-Window Math

Aggregation semantics define what happens between collection intervals. Most backends treat metrics as time series with a temporality and a set of aggregation functions.

• Temporality describes whether the backend expects cumulative values (typical for counters) or values per interval (typical for some gauge-like patterns).
• Bucket boundaries for histograms define how values are grouped. If your buckets are too coarse, p95 becomes a guess. If they’re too fine, you pay with storage and compute.
• Label dimensions determine how series are split. More label combinations means more series, which affects cost and query complexity.

The collector may also apply transformations, but the core semantics still come from the instrument type and the metric’s intended meaning.

Choosing the Right Instrument for Common Scenarios

Requests and Errors

Use a Counter for total requests and a Counter for total errors. Add a label like http.method and http.status_code to slice results.

Example reasoning: a request total should never decrease, even if you restart the process. If you restart, the counter resets; the backend can handle that when it knows the temporality and reset behavior.

In-Flight Work

Use an UpDownCounter for in-flight tasks if you record deltas at state transitions (start increments, finish decrements). This avoids sampling artifacts that happen when you only read “current” values.

Latency Distributions

Use a Histogram for request duration. Record the duration once per request. Choose buckets that match your SLO shape. For example, if most requests are under 200ms but you care about slow tails, include dense buckets in the low hundreds of milliseconds.

Current Queue Depth

Use a Gauge for queue depth if you can read it directly. If you only have events (enqueue/dequeue), you can also model it with an UpDownCounter, but then you’re aggregating deltas rather than reporting a direct measurement.

Example: Mapping Code Intent to Metric Semantics

Suppose you want to track HTTP request duration and total requests.

• Record duration with a Histogram so you can compute percentiles from bucketed counts.
• Record request totals with a Counter so you can compute rates over time.

Metric: http.server.duration
Instrument: Histogram
Record: observe(duration_ms) once per request
Aggregation: bucket counts over time windows

Metric: http.server.request_count
Instrument: Counter
Record: add(1) once per request
Aggregation: sum of increments over time windows

Example: Avoiding Semantic Mismatches

If you record request duration into a Gauge by setting it to the latest observed duration, you lose the distribution. A dashboard might still show a line, but it will represent “last sample,” not “typical latency.” Similarly, if you use a Counter for something that goes down, you’ll either produce negative increments that violate the intended meaning or force awkward workarounds.

Summary of the System

Instrument type answers: what kind of value are you recording? Aggregation semantics answer: how will the system combine those values over time and across label dimensions? When both match the metric’s real-world meaning, queries become straightforward and the numbers stay honest.

4.4 Designing Metric Names and Labels with Consistent Dimensions

Metric names and labels are how you turn raw measurements into something you can query without guessing. The goal is simple: every metric should be identifiable by its name, and every slice of that metric should be describable by a consistent set of dimensions.

Metric Names That Stay Stable

Use a metric name that describes what is being measured, not how it is computed. Prefer a pattern like http.server.duration or db.client.connections where the prefix groups related metrics and the suffix clarifies the unit or meaning.

Keep these rules practical:

• Use a consistent separator style across your system (commonly dots). Mixing styles makes dashboards brittle.
• Avoid embedding high-cardinality values in the name. If you need to distinguish by user_id, that belongs in a label, not the name.
• Choose one unit convention and stick to it. If you use seconds, always use seconds. If you use bytes, always use bytes.

A quick example: http.server.request.duration is better than request_duration_ms_by_endpoint because the name stays about the measurement, while the endpoint detail belongs in labels.

Labels That Form Predictable Dimensions

Labels (also called attributes or dimensions depending on the ecosystem) are the axes you group by. Consistency matters more than cleverness.

A label set should answer: “What are the dimensions that define a meaningful slice of this metric?” For HTTP server metrics, typical dimensions include:

• http.method
• http.route or http.target
• http.status_code
• service.name

Not every metric needs every dimension, but when a dimension exists, it should mean the same thing everywhere. For example, if http.route is a normalized route template in one place, don’t switch to raw paths in another.

Cardinality Control Without Losing Usefulness

High-cardinality labels (like user_id, request_id, or full URLs) explode the number of time series. The result is slower queries and higher storage costs.

A systematic approach:

1. List candidate labels you might want to filter by.
2. Mark which ones are stable (few values) versus explosive (many values).
3. Keep stable dimensions on the metric.
4. Move explosive details to logs or traces, where they are naturally tied to a single event.

Example: If you want to investigate a single failing user, don’t add user_id to http.server.errors. Instead, keep http.status_code and http.route on the metric, then use logs or traces to find the specific user.

Designing a Dimension Contract

Treat the label set as a contract between instrumentation and queries.

• Name labels consistently: http.status_code should always be numeric or always be string, but don’t mix.
• Use the same normalization strategy: route templates should be consistent across services.
• Document required versus optional labels: required labels are always present; optional labels may be absent for some code paths.

Here’s a concrete pattern for HTTP server request duration:

• Metric: http.server.duration
• Unit: seconds
• Labels: http.method, http.route, http.status_code

Then your queries become predictable: “Average duration for GET /checkout with 200 responses.” No detective work required.

Example: Two Metrics with Different Label Sets

Suppose you instrument both request counts and request durations.

Metric A: Request Count

• Name: http.server.request.count
• Labels: http.method, http.route, http.status_code

Metric B: Request Duration

• Name: http.server.request.duration
• Labels: http.method, http.route, http.status_code

Keeping the same label set across both metrics makes it easy to correlate “count spikes” with “latency changes.” If you later add a new label like deployment.environment, you can apply it consistently to both metrics.

Example: Normalization Choices That Prevent Confusing Data

If http.route sometimes contains /users/123 and other times contains /users/{id}, your dashboards will show fragmented series. The fix is to normalize at instrumentation time so http.route always uses the same template style.

A practical rule: if a label value can vary per request, it probably shouldn’t be a metric dimension.

Example: A Label Set Checklist

Before you ship a metric, verify:

• The metric name describes the measurement and unit.
• Each label is a dimension you will actually group by.
• No label values are request-unique.
• The meaning and normalization of each label are consistent across services.
• The label set supports the queries you already know you’ll write.

When these checks pass, your metrics become easy to reason about. When they don’t, you’ll end up with dashboards that look correct but behave like puzzles.

4.5 Example Metric Schemas for HTTP RPC Database and Messaging

This section turns semantic conventions into concrete metric schemas you can model consistently across services. The goal is simple: every metric should answer a predictable question when you look at it later—without needing tribal knowledge.

Metric Schema Building Blocks

A useful schema has five parts:

1. Metric identity: name and unit.
2. Aggregation intent: sum, count, gauge, histogram.
3. Dimensions: the labels you group by.
4. Resource context: service and deployment identity.
5. Event meaning: what exactly is being measured.

A practical rule: keep dimensions small and stable. If you wouldn’t want to group by it in a dashboard, it probably shouldn’t be a dimension.

HTTP Request Metrics Schema

Use HTTP metrics to answer: “How fast and how often are requests happening, and what outcomes do we see?”

Schema A: Request Latency Histogram

• Metric name: http.server.duration
• Instrument type: histogram
• Unit: milliseconds (or nanoseconds, but be consistent)
• Dimensions:
  • http.method (GET, POST)
  • http.route (templated route like /orders/{id})
  • http.status_code (200, 404, 500)
  • network.protocol (optional, e.g., http/1.1, http/2)
• Resource context:
  • service.name
  • service.namespace (if you use it)
  • deployment.environment (e.g., prod, staging)

Schema B: Request Count Counter

• Metric name: http.server.request_count
• Instrument type: counter
• Unit: 1
• Dimensions: same as latency, but you can omit network.protocol to reduce cardinality.

Example reasoning: http.route should be templated. If you use raw paths, cardinality explodes and dashboards become slow.

RPC Client and Server Metrics Schema

RPC metrics answer: “What is the behavior of remote calls, and which method is involved?”

Schema C: RPC Duration Histogram

• Metric name: rpc.client.duration
• Instrument type: histogram
• Unit: milliseconds
• Dimensions:
  • rpc.system (e.g., grpc)
  • rpc.service (service name in the RPC framework)
  • rpc.method (method name)
  • rpc.grpc.status_code or rpc.status_code depending on your stack

Schema D: RPC Request Count Counter

• Metric name: rpc.client.request_count
• Instrument type: counter
• Unit: 1
• Dimensions: same as duration.

Example reasoning: keep rpc.method stable and avoid including request-specific identifiers. If you need those, put them in logs, not metric dimensions.

Database Metrics Schema

Database metrics answer: “How often do queries run, and how long do they take?”

Schema E: Database Query Duration Histogram

• Metric name: db.client.query.duration
• Instrument type: histogram
• Unit: milliseconds
• Dimensions:
  • db.system (e.g., postgresql, mysql)
  • db.operation (e.g., SELECT, INSERT, UPDATE)
  • db.name (optional; include only if it’s low cardinality)

Schema F: Database Query Count Counter

• Metric name: db.client.query_count
• Instrument type: counter
• Unit: 1
• Dimensions: same as duration.

Example reasoning: avoid labeling by full SQL text. If you must distinguish query shapes, use an application-level operation label like db.operation or a precomputed query name.

Messaging Metrics Schema

Messaging metrics answer: “Are messages flowing, and what are the delivery and processing characteristics?”

Schema G: Messaging Consumer Processing Duration Histogram

• Metric name: messaging.consumer.duration
• Instrument type: histogram
• Unit: milliseconds
• Dimensions:
  • messaging.system (e.g., kafka, rabbitmq)
  • messaging.destination (topic or queue name)
  • messaging.operation (e.g., consume, process)
  • messaging.message_type (optional; only if bounded)

Schema H: Messaging Publish and Consume Count Counters

• Metric name: messaging.producer.publish_count

• Instrument type: counter

• Unit: 1

• Dimensions: messaging.system, messaging.destination

• Metric name: messaging.consumer.consume_count

• Instrument type: counter

• Unit: 1

• Dimensions: messaging.system, messaging.destination

Example reasoning: messaging.destination is usually safe because it’s limited by your infrastructure. If you include partition IDs or per-message keys, cardinality will grow without mercy.

Putting It Together with a Consistent Label Set

Across HTTP, RPC, DB, and messaging, you’ll get cleaner dashboards if you standardize resource identity and keep dimensions signal-specific.

Recommended baseline dimensions:

• service.name
• deployment.environment

Signal-specific dimensions:

• HTTP: http.method, http.route, http.status_code
• RPC: rpc.system, rpc.service, rpc.method, rpc.status_code
• DB: db.system, db.operation
• Messaging: messaging.system, messaging.destination

This separation makes it easy to compare services while still answering the right question for each telemetry signal.

5.1 Span Kinds and Their Meaning in Distributed Workflows

Span kinds tell you what role a span plays in a distributed interaction. They are not just labels for humans; they shape how backends interpret relationships between client and server activity, how latency is attributed, and how you reason about causality. In OpenTelemetry, the common span kinds are internal, server, client, producer, and consumer.

Span Kind Foundations

A span represents a unit of work. The kind answers: “Who initiated this work, and who received it?” That question becomes crucial when you stitch traces across services.

• Internal: Work that stays within one process. There is no network boundary or messaging handoff.
• Server: Work that handles an incoming request. The server span is typically created when a request arrives.
• Client: Work that initiates an outgoing request. The client span is typically created before sending.
• Producer: Work that publishes a message to a broker or stream.
• Consumer: Work that processes a received message.

A practical rule: if you can point to a boundary where another component receives something, you’re likely looking at client/server or producer/consumer.

How Span Kinds Affect Trace Reasoning

Consider a request from Service A to Service B. If A creates a client span and B creates a server span, most tracing systems can present them as a pair. That pairing helps you answer questions like “How long did B take after A sent the request?” and “Where did the time go: network, queueing, or handler logic?”

If you mark both sides as internal, the trace still exists, but the backend has fewer hints about interaction semantics. You may still correlate via trace context, yet the UI and analytics lose structure.

Example: HTTP Request Client and Server

Service A calls Service B over HTTP.

• In Service A, create a client span around the HTTP call.
• In Service B, create a server span around the request handler.

When trace context is propagated, the server span becomes a child (or otherwise causally linked) to the client span, depending on your instrumentation and sampling decisions.

A common mistake is to create only an internal span in Service A and rely on Service B’s server span alone. You’ll still see B’s work, but you lose the explicit “request initiated by A” timing.

Example: Messaging Producer and Consumer

Service C publishes an event to a queue.

• In Service C, create a producer span when publishing.
• In Service D, create a consumer span when the message is received and processed.

The message payload or headers carry trace context so that the consumer span can relate to the producer span. This is where span kinds prevent confusion: producer spans explain “time to publish,” while consumer spans explain “time to process.” If you mark both as internal, you blur the handoff and make queueing delays harder to interpret.

Example: Internal Spans Inside a Server Handler

Inside Service B’s server handler, you might do local work like parsing, validation, or database query orchestration.

Those operations are typically internal spans because they do not represent a boundary where another component receives control. You can still create additional spans for outgoing calls (client) or message publishing (producer), which keeps the trace readable and the attribution honest.

Advanced Detail: Choosing the Kind When There Is No Clear Boundary

Sometimes a “call” is not a network request, but it still hands off work to another component. If the handoff is within the same process and you control the execution, internal is usually correct. If another process or system receives the work—via HTTP, RPC, queue, or stream—use the corresponding client/server or producer/consumer kind.

When in doubt, ask what the receiving side would label as its role. The receiving side’s span kind is a strong indicator: a handler that processes an incoming request is server, and a handler that processes a delivered message is consumer.

Quick Checklist

• Outgoing request initiated by your code: client
• Incoming request handled by your code: server
• Publishing to a broker: producer
• Processing a delivered message: consumer
• Local work without a handoff: internal

Correct span kinds make traces easier to interpret because they preserve the shape of interactions, not just the existence of events.

5.2 Trace Context Identifiers and Correlation Behavior

Trace context is the set of identifiers that lets separate telemetry events line up as one story. In OpenTelemetry, the main identifiers are the trace id and span id, plus optional trace flags that describe sampling decisions. Correlation behavior is what you get when those identifiers are propagated correctly and interpreted consistently across services.

Core Identifiers and What They Mean

A trace id identifies the end-to-end request flow across many services. A span id identifies a single operation within that flow, such as “handle HTTP request” or “query database.” A span also carries a parent relationship, which is how you reconstruct a tree of operations.

Correlation depends on two rules:

1. Every span belongs to exactly one trace id. If a span has a different trace id, it is a different story.
2. Every span (except the root) has a parent span id. If the parent link is missing or wrong, the tree shape breaks even if the trace id matches.

Trace Flags and Sampling Behavior

Trace flags indicate whether the trace is sampled. When sampling is off, you still want correlation identifiers to travel so downstream systems can make consistent decisions. In practice, you may see fewer spans exported, but the identifiers still explain why the trace looks sparse.

A common operational mistake is treating “not exported” as “not correlated.” Correlation is about identifiers; export is about policy. Keep those concepts separate when debugging.

Correlation Across Boundaries

Correlation behavior is easiest to understand by following identifiers across boundaries:

• Inbound request to service: the service extracts trace context from incoming headers, then creates a new span whose parent is the extracted context.
• Outbound call to another service: the service injects the current span context into outgoing headers, so the next service can attach its spans to the same trace.
• Async work via messaging: the producer injects context into message metadata; the consumer extracts it when creating spans for the processing work.

If you miss injection on outbound calls, the next service will start a new trace. If you miss extraction on inbound calls, your spans will have no parent relationship, even if the trace id might still appear.

Example: Synchronous HTTP Call Chain

Consider Service A handling an HTTP request and calling Service B.

• Service A receives the request with headers containing a trace id T1 and span context S0.
• Service A creates span S1 for request handling. Its parent is S0, and its trace id is T1.
• Service A calls Service B and injects the current context (trace id T1, span id S1) into outgoing headers.
• Service B extracts the context and creates span S2 for its handler. Its parent is S1, and its trace id is T1.

The correlation behavior you should see in a trace view is a single trace with a parent-child chain: S0 -> S1 -> S2.

Example: What Goes Wrong When Propagation Fails

If Service A forgets to inject context into the outgoing call, Service B will extract nothing and start a new trace.

• Service A spans: trace id T1
• Service B spans: trace id T2

Even if both services record similar attributes like http.route, the trace graphs will not connect. This is why trace id mismatches are the first thing to check when correlation looks “split.”

Example: Parent Link Breaks Without Trace Id Mismatch

A subtler failure happens when you reuse the trace id but lose the parent relationship. For example, you might create a span in Service B that uses T1 but sets its parent incorrectly (or as root). The result is still one trace, but the tree becomes flat or oddly rooted.

When debugging, verify both:

• Trace id equality across services
• Parent chain correctness for the spans that should be nested

Practical Debugging Checklist

When correlation behavior looks off, use this order:

1. Confirm the trace id is identical across the expected services.
2. Confirm the parent relationship exists where you expect a child span.
3. Check whether sampling flags explain missing spans rather than missing identifiers.
4. For async flows, verify that message metadata carries the context and that the consumer extracts it before creating spans.

This approach keeps you grounded in identifiers and relationships, which is where correlation issues actually live.

5.3 Span Attributes for HTTP RPC Database and Messaging

Span attributes are the small, structured facts that make spans searchable, comparable, and explainable. For HTTP, RPC, database calls, and messaging, the goal is consistent attribute keys and values that let you answer questions like “Which endpoint was slow?”, “Which query shape caused the spike?”, or “Which queue hop failed?”. The trick is to treat attributes as a contract: instrumentation should emit them with stable meaning, and collectors should preserve them end to end.

Foundational Rules for Attribute Quality

Start with four practical rules.

1. Use semantic keys, not ad-hoc names. If an attribute has a defined meaning in the semantic conventions, prefer it. This keeps dashboards and alerts from turning into a scavenger hunt.
2. Choose the right granularity. Put high-cardinality values (like full URLs with query strings) behind safer fields or normalize them. A span attribute can be precise without being explosive.
3. Keep types consistent. Numbers should be numbers, booleans should be booleans, and strings should be strings. Mixed types break aggregations.
4. Record both identity and outcome. Identity attributes tell you what happened; status and error attributes tell you how it went.

HTTP Span Attributes for Requests and Responses

For HTTP, the most useful attributes describe the request target, method, route, and response outcome.

• Request identity: method, route (or template), and target details.
• Client/server role: span kind distinguishes whether you’re observing an inbound request or an outbound call.
• Outcome: HTTP status code and any error indicator.

A common best practice is to capture route rather than the raw path when your framework can provide it. Route values typically have lower cardinality and remain stable across deployments.

Example span attribute set for an inbound request:

• http.method: GET
• http.route: /api/orders/{id}
• http.target: /api/orders/123 (optional, use carefully)
• http.status_code: 200
• net.peer.name: client.example.com (if available)

Example for an outbound call:

• http.method: POST
• http.route: /payments/charge
• http.status_code: 503
• http.flavor: 1.1 (if your instrumentation exposes it)

RPC Span Attributes for Service-to-Service Calls

RPC attributes mirror HTTP’s identity and outcome, but they focus on procedure calls rather than URLs.

• System and service identity: the RPC system (for example, gRPC), the service name, and the method.
• Peer identity: the remote host or service endpoint.
• Outcome: status code or equivalent error mapping.

If you instrument both client and server sides, ensure the client span and server span agree on the method identity fields. That alignment is what makes cross-service latency analysis actually work.

Example RPC attributes:

• rpc.system: grpc
• rpc.service: CheckoutService
• rpc.method: Charge
• rpc.grpc.status_code: UNAVAILABLE
• net.peer.name: checkout.internal

Database Span Attributes for Queries and Commands

Database spans should capture what was executed and how it behaved, without dumping entire SQL strings into every span. The semantic conventions support both query identity and safe query text.

Key ideas:

• Database identity: system, name, and user context when available.
• Statement identity: operation name and a normalized statement or query template.
• Outcome: database system status and error mapping.

A practical approach is to record:

• db.system: postgresql
• db.name: orders
• db.operation: SELECT
• db.sql.table: orders (if your instrumentation can infer it)
• db.statement: SELECT * FROM orders WHERE id = $1 (parameterized form)
• db.user: app_user (optional)

If you only have raw SQL, consider truncation and parameterization so that spans remain searchable without turning into a log of sensitive data.

Messaging Span Attributes for Queue and Topic Hops

Messaging spans should explain where a message came from, where it went, and what happened to it.

• Messaging system: the broker type (for example, kafka, rabbitmq).
• Destination identity: topic or queue name.
• Message identity: message id when available, and sometimes a correlation key.
• Outcome: publish/consume success and error mapping.

Example producer span attributes:

• messaging.system: kafka
• messaging.destination: orders.events
• messaging.destination_kind: topic
• messaging.operation: publish
• messaging.message_id: evt_9f3a...

Example consumer span attributes:

• messaging.system: kafka
• messaging.destination: orders.events
• messaging.destination_kind: topic
• messaging.operation: process
• messaging.message_id: evt_9f3a...
• messaging.kafka.consumer.group: order-service

Mind Map: Span Attribute Coverage

Putting It Together in One Span

A single request often touches multiple systems: an inbound HTTP request triggers an RPC call, which executes a database query, which then publishes a message. The best practice is to keep each span’s attributes focused on its own work, while relying on trace context to connect them.

For example, an inbound GET /api/orders/{id} span might carry HTTP identity and status, while the child database span carries db.* fields and the messaging span carries messaging.* fields. This separation keeps queries readable and prevents attribute collisions.

Finally, verify attribute presence and consistency with a simple checklist: each span should have (1) the semantic “system” identity (HTTP/RPC/db/messaging), (2) a destination or route identity, and (3) an outcome field via span status and any protocol-specific status code.

5.4 Event and Status Modeling for Error and Outcome Reporting

Event and status modeling answers two practical questions: “What happened?” and “How should a user interpret it?” In OpenTelemetry, you typically express these with a combination of span status, span events, and attributes. The goal is to make error analysis possible without forcing every downstream system to guess.

Span Status as the Coarse Outcome

Span status is the coarse-grained outcome of the operation represented by the span. Use it to communicate success versus failure in a way that is consistent across services.

• Status code: Prefer setting status to OK for successful operations and to an error code for failures.
• Status message: Add a short, human-readable explanation when it helps triage. Keep it stable enough that dashboards and alerts can rely on it.
• When to set status: Set status when the span’s operation completes. If you create intermediate events (like retries), keep status focused on the final outcome.

A common pattern is: record retries as events, then set final status once you know whether the request ultimately succeeded.

Span Events as the Fine-Grained Timeline

Span events are timestamped annotations attached to a span. They are ideal for capturing meaningful moments that are too detailed for status.

Use events for:

• Retry attempts and backoff decisions
• External dependency calls and their outcomes
• Validation failures that still allow the request to continue
• Business-relevant milestones like “payment authorized” or “order persisted”

Use span events carefully:

• Don’t emit an event for every log line. Events should be queryable milestones, not a duplicate of application logging.
• Keep event names consistent and low-cardinality. If you need high-cardinality data, store it as attributes rather than inventing new event names.

Attribute Design for Error and Outcome Reporting

Attributes provide the structured details that make events and status actionable. A good attribute strategy has three layers.

1. Outcome attributes: What was the result? Examples include http.status_code, db.operation, or messaging.system.
2. Error attributes: What failed and why? Examples include error.type, error.code, and error.message.
3. Context attributes: Where and under what conditions? Examples include retry_count, timeout_ms, component, and customer_segment.

Keep attribute values consistent across services. If one service uses error.code and another uses failure_reason, you’ll spend time normalizing later.

Event Naming and Attribute Conventions

Event names should be verbs or noun-phrases that describe the moment. Examples:

• retry_scheduled
• dependency_call_started
• dependency_call_failed
• validation_failed
• fallback_used

Attributes on those events should follow a predictable schema. For instance, a dependency_call_failed event should include:

• dependency.name
• dependency.system
• dependency.operation
• error.type
• error.code
• error.message

This makes it possible to filter for “all dependency failures” without knowing every service’s internal wording.

Systematic Modeling Workflow

A reliable workflow keeps teams aligned:

1. Define outcomes: Decide what counts as success versus failure for the span.
2. Map failures to status: Set span status based on the final outcome.
3. Emit events for milestones: Add events for retries, dependency outcomes, and key decision points.
4. Attach attributes for triage: Include error classification and relevant context.
5. Validate cardinality: Ensure event names are stable and attribute values don’t explode in uniqueness.

Example: HTTP Request with Retries and Final Failure

Below is a compact example of how status and events work together. The span ends with failure status, while events capture the retry timeline.

Span: GET /checkout
Status: ERROR
Status message: "request failed after retries"

Event: dependency_call_failed
  dependency.system: "http"
  dependency.name: "payment-service"
  dependency.operation: "POST /authorize"
  error.type: "timeout"
  error.code: "DEADLINE_EXCEEDED"
  error.message: "upstream did not respond in time"

Event: retry_scheduled
  retry_count: 1
  backoff_ms: 200

Event: dependency_call_failed
  dependency.system: "http"
  dependency.name: "payment-service"
  dependency.operation: "POST /authorize"
  error.type: "timeout"
  error.code: "DEADLINE_EXCEEDED"
  error.message: "upstream did not respond in time"

Attributes on span:
  http.status_code: 504
  retry_count: 2
  timeout_ms: 500

This structure supports two queries: one for the final outcome (http.status_code and span status) and another for the timeline (dependency_call_failed events and their error codes).

Example: Business Outcome Without Technical Failure

Not every “bad outcome” is a technical error. Suppose an order is rejected due to insufficient funds. The operation may complete successfully from an infrastructure perspective.

Model it like this:

• Span status: OK because the request was handled.
• Span event: order_rejected with attributes like rejection.reason and rejection.code.
• Optional span attributes: order.state.

This prevents alerting systems from treating business rejections as infrastructure incidents, while still keeping the outcome searchable.

Example: Technical Failure with Partial Success

Sometimes part of a workflow succeeds and part fails. Use events to show what succeeded, then set status based on the overall operation semantics.

For example, a batch job might process 90 items and fail on the rest. Emit events like item_processed for key items (not every single one), then set span status to error if the batch is considered failed. Include attributes such as processed_count and failed_count so triage can happen without scanning every event.

5.5 Example Trace Templates for End-to-End Request Flows

End-to-end request flows are easiest to reason about when you treat a trace as a structured story: one entry span, a chain of causally related spans, and a clear outcome. The templates below are designed to map cleanly to OpenTelemetry span kinds, semantic attributes, and context propagation rules.

Template 1: HTTP Request Through Service to Database

Use this when a client calls your service over HTTP and your service queries a database.

Span layout

• HTTP SERVER span: represents the inbound request.
• DB CLIENT span: represents the database query.
• Optional INTERNAL span: represents business logic that orchestrates work.

SERVER span attributes (example)

• http.method: GET
• http.route: /v1/orders/{orderId}
• http.target: /v1/orders/123
• http.status_code: 200
• net.peer.ip and net.peer.port when available
• service.name and service.instance.id from resource attributes

DB CLIENT span attributes (example)

• db.system: postgresql
• db.name: orders
• db.operation: SELECT
• db.sql.table: orders when you can infer it safely
• db.statement: include only if your policy allows it; otherwise omit

Outcome and errors

• If the database fails, mark the DB span ERROR and add an exception event.
• Mark the SERVER span ERROR only if the request ultimately fails; otherwise keep it OK and record a domain outcome attribute like event.outcome or an application-level order.lookup result.

Example trace flow

• Client sends request with traceparent.
• Your service creates a SERVER span with that trace context.
• Your service starts a DB CLIENT span as a child span.
• The SERVER span ends after the response is written.

Template 2: HTTP Request with Fan-Out to Another Service

Use this when your service calls a downstream HTTP service and you want a single trace to show the full latency breakdown.

Span layout

• HTTP SERVER span (entry)
• INTERNAL span for orchestration (optional but helpful)
• HTTP CLIENT span for the downstream call
• Optional INTERNAL span for response transformation

Propagation details

• The downstream HTTP CLIENT span must inject the current trace context into outgoing headers.
• The downstream service’s SERVER span should become a child of the CLIENT span automatically.

Attribute consistency rules

• Use the same service.name values across services.
• For the CLIENT span, set http.method, http.status_code (when known), and target fields like http.url or http.target depending on what your instrumentation provides.

Error modeling

• If the downstream returns 500, set the CLIENT span status to ERROR and record http.status_code.
• Decide whether the SERVER span is ERROR based on whether you return a failure to the caller.

Template 3: Messaging Flow from Producer to Consumer

Use this when work is triggered by publishing a message and completed by a consumer.

Span layout

• PRODUCER span: created when publishing
• CONSUMER span: created when processing
• Optional INTERNAL spans inside the consumer for business logic

Propagation details

• Inject trace context into message headers.
• The consumer extracts context and starts the CONSUMER span with the extracted parent.

Attribute modeling

• messaging.system: e.g., kafka
• messaging.destination: topic or queue name
• messaging.operation: publish or process depending on your semantic mapping

Error modeling

• If processing fails, mark the CONSUMER span ERROR and add an exception event.
• If you retry, keep the trace consistent by continuing the same trace context for each attempt, or record attempt counts as attributes on the consumer span.

Verification Checklist for Templates

• Parent-child relationships match causality: entry span is root, outbound CLIENT spans are children, and consumer spans link via extracted context.
• Each span has a clear kind: SERVER for inbound, CLIENT for outbound, PRODUCER/CONSUMER for messaging.
• Outcome is unambiguous: status and exception events align with what the caller experiences.
• Semantic attributes are present where they matter: HTTP method and route, DB system and operation, messaging system and destination.

Minimal Example Trace Shape

Trace
- Span A HTTP SERVER GET /v1/orders/{orderId} status=OK
  - Span B INTERNAL orchestrate lookup
    - Span C DB CLIENT SELECT orders status=OK
  - Span D HTTP CLIENT POST /v1/pricing status=OK

This shape is intentionally boring: it’s the kind of structure that stays readable when you add real attributes, real errors, and real latency.

6.1 Log Record Structure and Field Mapping to Telemetry Concepts

A log record is a single, time-stamped event emitted by an application or component. In OpenTelemetry terms, the goal is not just to “send text,” but to attach structured fields that can be filtered, correlated, and compared across services. Think of a log record as a small packet: it has a body (what happened), a time (when), and metadata (who, where, and how to interpret it).

Core Fields in a Log Record

Start with the minimum set that makes a log useful in practice:

• Timestamp: The moment the event occurred. Use the event time, not the export time, so ordering and latency analysis remain meaningful.
• Severity: A level such as debug, info, warn, error. Severity drives alerting and dashboards, so keep it consistent across services.
• Body: The main message content. Prefer structured bodies (or consistent message templates) so queries don’t rely on brittle string matching.
• Attributes: Key-value pairs that carry context. These are the workhorses for correlation and filtering.
• Trace Context: Optional identifiers that link the log to a trace and span.

A practical rule: if a field changes the meaning of the event, it belongs in attributes. If it only helps humans read the message, it can stay in the body.

Mapping Log Fields to Telemetry Concepts

OpenTelemetry encourages a consistent mental model across signals. Logs map cleanly when you treat them as “events with context,” similar to trace events and metric dimensions.

• Service Identity: Put service name, namespace, and instance-like details in resource attributes. This mirrors how traces and metrics identify the emitting component.
• Event Semantics: Use attributes to represent what the log is about (for example, operation, endpoint, queue name). This parallels span attributes that describe the work.
• Outcome and Error Modeling: Represent outcomes with attributes like event.outcome and error details with exception.* fields when available. This keeps error analysis uniform.
• Correlation: Attach trace_id and span_id (or equivalent trace context fields) so logs can be pulled into the same request view as traces.

Example Log Record with Reasoned Field Choices

Imagine an API gateway handling a request and failing to reach an upstream service. A good log record separates concerns:

• Body: A concise message like “Upstream call failed.”
• Severity: error because the request outcome is degraded or failed.
• Attributes:
  • Request context: http.method, http.route, http.status_code.
  • Upstream context: net.peer.name, rpc.system, rpc.service.
  • Error context: exception.type, exception.message.
  • Correlation: trace_id and span_id so the log appears in the trace timeline.
• Resource attributes: service.name for the gateway, plus environment and region.

This structure supports three common queries without guesswork: “show all upstream failures for route X,” “show errors for service Y in region Z,” and “for this trace, show the related logs.”

Practical Field Mapping Checklist

Use this checklist to avoid the two most common problems: missing context and inconsistent naming.

1. Put identity in resource attributes: service and deployment details should not be repeated in every log attribute.
2. Put event meaning in attributes: route, operation, destination, and outcome belong in attributes.
3. Keep severity aligned with outcomes: don’t log an error as info just because the message is informative.
4. Attach trace context when available: if you already have a span, link the log to it.
5. Avoid attribute sprawl: only add fields that you will filter on or use for correlation.

Example Attribute Set for a Single Failure Event

{
  "timestamp": "2026-03-25T10:15:30.123Z",
  "severity": "error",
  "body": "Upstream call failed",
  "attributes": {
    "http.method": "GET",
    "http.route": "/checkout",
    "http.status_code": 502,
    "rpc.system": "grpc",
    "rpc.service": "payments",
    "rpc.method": "Charge",
    "event.outcome": "failure",
    "exception.type": "Unavailable",
    "exception.message": "upstream timeout",
    "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
    "span_id": "00f067aa0ba902b7"
  },
  "resource": {
    "service.name": "api-gateway",
    "deployment.environment": "prod",
    "cloud.region": "us-east-1"
  }
}

The key is consistency: the same attribute names should appear across services for the same concepts, so collectors and backends can treat logs as structured data rather than plain text with opinions.

6.2 Required and Recommended Attributes for Vendor Neutral Logs

Vendor neutral logs work when every record carries enough context to be grouped, filtered, and correlated—without relying on backend-specific field names. OpenTelemetry’s log data model uses a consistent split between resource attributes (who produced the telemetry) and log record attributes (what happened). The goal is simple: make the same log query work across services and backends.

Core Idea: Separate Identity from Event Details

A log record should answer three questions:

1. Where did this come from? Use resource attributes.
2. What is the event? Use log record attributes.
3. How do I connect it to other telemetry? Use correlation fields like trace identifiers.

When you keep this separation, you avoid the common trap of stuffing everything into one namespace and then discovering half your fields are missing in other services.

Required Attributes for Practical Interoperability

In OpenTelemetry, the “required” set is best understood as the minimum fields that make a log record meaningful and queryable.

• Timestamp: The time the event occurred (not the time it was exported). If you only have “now,” you still need to be consistent about what “now” means.
• Severity number and severity text: Severity enables filtering like “show me errors” without parsing message strings.
• Body: The human-readable message or structured payload that describes the event.
• Resource attributes: At minimum, identify the service and environment so logs from different producers don’t collapse into a single undifferentiated blob.

A useful rule of thumb: if you can’t write a query that isolates one service’s error logs in under a minute, your required attributes aren’t doing their job.

Recommended Attributes That Prevent Query Pain

Recommended attributes are where vendor neutrality becomes real. They standardize the dimensions you’ll want later.

• Service identity: service.name and service.namespace (when applicable) plus service.version if you deploy multiple versions.
• Deployment environment: deployment.environment such as production, staging, or dev.
• Host and runtime context: host.name and os.type when you need to distinguish node-level issues.
• Thread or execution context: process.thread.name or similar fields when concurrency matters.
• Request correlation: trace_id and span_id when the log is tied to a specific span.
• HTTP and RPC details: http.method, http.target, http.status_code, rpc.system, rpc.service, rpc.method when the event is request-scoped.
• Error details: exception.type, exception.message, and exception.stacktrace when an error occurs.

These recommendations are not about being thorough for its own sake. They’re about making the same filters and groupings work across services.

Example: Request-Scoped Error Log with Correlation

{
  "resource": {
    "service.name": "checkout-api",
    "deployment.environment": "production",
    "service.version": "1.18.3",
    "host.name": "checkout-7f9c2"
  },
  "timestamp": "2026-03-25T10:15:30.123Z",
  "severity": {"number": 17, "text": "ERROR"},
  "body": "Payment authorization failed",
  "attributes": {
    "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
    "span_id": "00f067aa0ba902b7",
    "http.method": "POST",
    "http.target": "/v1/checkout",
    "http.status_code": 402,
    "exception.type": "PaymentDeclined",
    "exception.message": "Issuer declined authorization",
    "exception.stacktrace": "..."
  }
}

This record is easy to use because it supports three common queries: “errors by service,” “errors by endpoint,” and “errors tied to a specific trace.”

Example: Background Job Log Without HTTP Fields

{
  "resource": {
    "service.name": "order-processor",
    "deployment.environment": "production",
    "service.version": "2.4.0"
  },
  "timestamp": "2026-03-25T10:16:05.500Z",
  "severity": {"number": 9, "text": "INFO"},
  "body": "Requeued message due to transient database timeout",
  "attributes": {
    "exception.type": "TimeoutError",
    "exception.message": "DB query exceeded 2s",
    "process.thread.name": "worker-3"
  }
}

Notice what’s missing: there’s no http.* because the event isn’t request-scoped. Vendor neutrality isn’t about forcing every field everywhere; it’s about using consistent fields when they apply.

Practical Checklist for Consistent Log Attributes

• Always set timestamp, severity, and body.
• Always set service identity and deployment environment in resource attributes.
• Add trace correlation when the log is produced inside a span.
• Add request context only when the event is tied to HTTP or RPC.
• Add exception fields when you’re reporting an error condition.

When these rules are followed, your logs stay readable to humans and usable to machines—without requiring backend-specific gymnastics.

6.3 Correlating Logs with Traces Using Trace Identifiers

Correlation is the practical bridge between “what happened” (logs) and “how it happened across services” (traces). The goal is simple: every log record emitted during a request should carry the same trace identity used by the trace spans for that request. When you do this consistently, you can start from a log line and jump to the exact distributed execution path.

Core Concepts for Trace-Based Log Correlation

A trace is identified by a trace_id, and a specific span within that trace is identified by a span_id. For log correlation, the minimum useful set is trace_id. Many teams also include span_id so they can pinpoint which step produced the log.

In OpenTelemetry, these identifiers are typically exposed to logging code through the active context. That means your logger doesn’t need to know anything about HTTP headers or messaging protocols; it just needs to read the current context and copy the identifiers into the log record.

Step-by-Step: From Request to Correlated Log

1. A request enters the service (for example, via HTTP). Instrumentation creates a span for the request and stores it in the active context.
2. Your code emits logs while the span is active. A log enrichment mechanism reads trace_id (and optionally span_id) from the active context and attaches them to the log record.
3. The log record is exported through OTLP. The identifiers travel with the log payload.
4. The backend indexes the fields so you can query logs by trace_id and immediately find the matching trace.

This flow matters because it avoids brittle manual wiring. If you rely on the active context, you get correlation “for free” across most code paths.

Example: Logging with Trace and Span Identifiers

Below is a conceptual example showing how a logger can attach identifiers from the current context. The exact APIs differ by language, but the idea is consistent.

function logWithCorrelation(level, message, fields):
  ctx = getActiveContext()
  span = getSpanFromContext(ctx)
  if span is not null:
    fields["trace_id"] = span.traceId
    fields["span_id"] = span.spanId
  fields["service.name"] = getServiceName()
  emitLog(level, message, fields)

If you also include service.name and a stable event.name (or similar field), your log queries become more precise than “trace_id only,” especially when you’re investigating multi-service workflows.

Example: Correlating a Log Line to a Trace

Imagine a log line like this:

• message: payment declined
• trace_id: 4bf92f3577b34da6a3ce929d0e0e4736
• span_id: 9f2c1b2a3d4e5f60
• service.name: checkout-api

In the backend, you filter logs by trace_id = 4bf92f3577b34da6a3ce929d0e0e4736. Then you pivot to the trace view for the same trace_id. The span_id helps you land on the span that produced the log, which is usually the request handler span or a child span representing the payment call.

Advanced Details That Prevent “Almost Correlated” Data

1. Ensure context propagation is actually happening. If the incoming request lacks trace context, your service will start a new trace and logs will correlate to that new trace. That’s correct behavior, but it can surprise you if you expected correlation with an upstream trace.

2. Don’t log after the context is gone. If you emit logs in background tasks after the request scope ends, the active context may be empty. In that case, either capture the trace context at the time you schedule the work or explicitly pass the identifiers into the background task.

3. Avoid overwriting identifiers in processors. Collector processors can enrich or transform fields. If a processor accidentally replaces trace_id with a different value, correlation breaks silently. Treat trace_id as a protected join key.

4. Keep field names consistent across signals. Your backend’s correlation features often assume specific field names and formats. Use the same conventions for trace_id and span_id across all services so queries work uniformly.

Mind Map: Query Patterns That Use Trace Identifiers

Practical Checklist for Reliable Correlation

• Every request-scoped log includes trace_id.
• Optional: include span_id for precise origin.
• Background work captures context before leaving the request scope.
• Collector processors do not modify trace_id or span_id.
• Backend indexing supports fast filtering by trace_id.

When these are in place, correlation stops being a scavenger hunt and becomes a deterministic join: one identifier, two views, and a clear path from a single log line to the full distributed execution.

9. Collector Pipelines for Metrics Logs and Traces

9.1 Collector Pipeline Anatomy Receivers Processors Exporters

A collector pipeline is a controlled conveyor belt for telemetry. Each stage has a job, and the collector keeps the stages separate so you can reason about what happens to data at each step.

Pipeline Stages and Their Responsibilities

Receivers accept telemetry from clients or other collectors. They define the network surface and the initial decoding rules. If a receiver can’t parse a payload, nothing else in the pipeline gets a chance to help.

Processors transform, filter, enrich, or batch data. They run after decoding and before export, so they can rely on a consistent internal representation of traces, metrics, and logs.

Exporters send the processed telemetry to a destination such as a backend, another collector, or a storage system. Exporters are where delivery semantics show up: retries, timeouts, and how failures are handled.

A useful mental model is: receive to normalize, process to shape, export to deliver.

Mind Map of the Pipeline Flow

- Collector Pipeline - Receivers - OTLP gRPC and HTTP - Prometheus scrape - File log tailing - Health and metrics endpoints - Processors - Resource and attribute enrichment - Batching and queueing - Filtering and dropping - Transformation and normalization - Sampling and aggregation adjustments - Exporters - OTLP to backend - Logging for debugging - Storage or analytics sinks - Retry and backoff behavior - Cross-Cutting Concerns - Ordering within a pipeline - Backpressure and buffering - Error handling and observability of the collector

How Data Moves Through a Pipeline

Within a single pipeline, processors run in the order they are configured. That order matters. For example, if you filter out spans based on an attribute, do it after the attribute is added; otherwise the filter never matches.

Also note that batching is not just a performance tweak. It changes the timing of when data becomes visible downstream, which affects alerting and dashboards that expect near-real-time updates.

Example Pipeline Configuration Skeleton

Below is a compact example showing the anatomy for traces. The same pattern applies to metrics and logs.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, attributes, tail_sampling]
      exporters: [otlp]

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 1s
  attributes:
    actions:
      - key: service.name
        action: insert
        value: example-service
  tail_sampling:
    decision_wait: 5s

exporters:
  otlp:
    endpoint: https://backend.example:4317
    tls:
      insecure: false

The key idea is that the pipeline section wires stages together, while each stage section defines behavior.

Practical Examples of Stage Responsibilities

Receiver example: OTLP endpoint hygiene
If you expose OTLP on both gRPC and HTTP, keep the receiver configuration explicit. That prevents accidental protocol mismatches when clients switch libraries.

Processor example: attribute enrichment before filtering
Suppose you want to drop spans from a noisy endpoint. If the endpoint attribute is not present until after enrichment, place enrichment before filtering. Otherwise you’ll drop nothing and wonder why the noise remains.

Exporter example: delivery behavior and failure visibility
When an exporter can’t reach the backend, the collector’s retry and buffering determine whether you lose data or delay it. You should treat exporter configuration as part of your reliability story, not as a final checkbox.

Systematic Debugging Using Stage Boundaries

When telemetry is missing, don’t start by staring at the backend. Instead, test each stage boundary:

1. Receiver check: confirm the collector is receiving and decoding the payload.
2. Processor check: confirm transformations and filters are applied as intended.
3. Exporter check: confirm the destination is reachable and accepts the data.

A simple way to do this is to temporarily add a debug exporter that prints a small sample. If the data appears there but not in the backend, the issue is in export or downstream ingestion.

Mind Map of Common Ordering Pitfalls

- Ordering Pitfalls - Filter before Enrich - Symptom: filter matches nothing - Batch before Transform - Symptom: transform sees unexpected grouping - Sampling after Drop - Symptom: sampling has no effect - Exporter too early - Symptom: transformations not applied - Multiple pipelines confusion - Symptom: data routed to wrong signal

Integrated Takeaway

A collector pipeline is easiest to reason about when you treat it as a sequence of contracts: receivers guarantee decoding, processors guarantee shaping, and exporters guarantee delivery attempts. Once you respect those contracts, configuration becomes less guesswork and more engineering.

9.2 Building Separate Pipelines for Each Signal Type

Separate pipelines keep each signal’s processing rules clear, testable, and cheaper to operate. In the OpenTelemetry Collector, a pipeline is a named route from one or more receivers through optional processors to one or more exporters. When you split metrics, logs, and traces, you avoid “one size fits none” transformations and you can tune batching, retry behavior, and attribute normalization per signal.

Why Separate Pipelines Work

A pipeline boundary is a practical contract: everything inside it assumes a specific data model. Metrics processors can safely rewrite metric names or transform aggregation temporality without touching trace spans. Log processors can enrich records with correlation fields without risking span status semantics. Traces can apply sampling or span attribute normalization without accidentally dropping log fields.

A common operational win is isolating backpressure. If your trace backend slows down, you can keep metrics flowing by using different batch sizes and retry settings per pipeline.

Mind Map: Pipeline Separation Strategy

- Separate Pipelines for Each Signal Type - Goal - Clear processing rules per data model - Independent tuning for batching retry and backpressure - Collector Structure - Receivers - OTLP receiver shared entry point - Pipelines - metrics pipeline - logs pipeline - traces pipeline - Processors - metrics processors - logs processors - traces processors - Exporters - metrics exporter - logs exporter - traces exporter - Design Steps - Identify required transformations per signal - Choose minimal processors per pipeline - Align resource attributes and service identity - Validate with signal-specific checks - Operational Checks - Verify counts per signal - Inspect dropped items and processor errors - Confirm attribute presence and naming consistency

Step-by-Step Pipeline Design

1. Start with a shared receiver: Use a single OTLP receiver to accept all signals. This keeps network and authentication configuration in one place.
2. Create one pipeline per signal: Define metrics, logs, and traces pipelines even if they initially look similar.
3. Add processors only where they make sense: For example, metric transformation belongs in the metrics pipeline; log body parsing belongs in the logs pipeline; span filtering belongs in the traces pipeline.
4. Export with signal-specific destinations: Many teams route traces and logs to different backends or different indices.
5. Validate per pipeline: Confirm that each pipeline exports the expected fields and that processor errors are visible.

Example Collector Configuration Skeleton

This skeleton shows the separation pattern. It uses one OTLP receiver and three pipelines with different processor sets.

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

  # Placeholder Processors You Would Tailor per Signal
  metrics_transform: {}
  log_enrich: {}
  trace_filter: {}

exporters:
  otlphttp/metrics:
    endpoint: https://example-metrics
  otlphttp/logs:
    endpoint: https://example-logs
  otlphttp/traces:
    endpoint: https://example-traces

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [metrics_transform, batch]
      exporters: [otlphttp/metrics]

    logs:
      receivers: [otlp]
      processors: [log_enrich, batch]
      exporters: [otlphttp/logs]

    traces:
      receivers: [otlp]
      processors: [trace_filter, batch]
      exporters: [otlphttp/traces]

Concrete Processor Choices That Benefit from Separation

Metrics pipeline: Use metric-specific processors to normalize names and labels. For instance, if your application emits http.server.duration and http.duration_ms, you can map both to a single semantic name and ensure consistent units. This prevents dashboards from splitting by accident.

Logs pipeline: Enrich logs with stable correlation fields. A typical pattern is to ensure every log record includes service.name and a trace identifier field when available. If you parse structured log bodies, do it here so you don’t risk breaking trace attribute expectations.

Traces pipeline: Apply span filtering or attribute cleanup based on span kind and status. For example, you might drop spans that represent internal health checks, while keeping spans that represent external calls. Doing this only in the traces pipeline avoids accidentally removing log events that describe those checks.

Validation Checklist per Pipeline

• Metrics: Confirm metric names and label keys match your semantic conventions and that units are consistent.
• Logs: Confirm the log body and parsed fields are present, and correlation fields appear when trace context exists.
• Traces: Confirm span relationships are intact, and that any filtering doesn’t remove critical parent spans.

A practical test is to send one synthetic request that produces one trace, one log event, and one metric update. Then verify that each pipeline exports exactly the expected count and that the shared resource identity fields match across all three signals.

9.3 Routing and Fan Out Strategies for Multi Destination Delivery

Routing decides where telemetry goes; fan out decides how many places it goes to. In the OpenTelemetry Collector, both happen inside pipelines: routing is typically implemented with processors that inspect attributes, and fan out is implemented by sending the same processed data to multiple exporters.

Core Idea: Route by Attributes, Fan Out by Intent

Start with a simple rule: use resource attributes for “who is this?” and span/log attributes for “what is this?”. For example, route by service.name and deployment.environment, then fan out by signal type.

A practical baseline is:

• Metrics: one primary backend for dashboards, plus an optional secondary for long-term retention.
• Traces: one backend for trace exploration, plus a second backend for compliance sampling.
• Logs: one backend for search, plus an optional sink for alerting pipelines.

This keeps each destination’s strengths aligned with the data it receives, without duplicating everything everywhere.

Mind Map: Routing and Fan Out in Collector Pipelines

# Routing and Fan Out Strategies - Inputs - OTLP receivers - Multiple endpoints - Routing Decisions - Resource attributes - service.name - deployment.environment - cloud.region - Telemetry attributes - http.route - rpc.system - messaging.destination - Fan Out Mechanisms - Multiple exporters per pipeline - Signal-specific pipelines - Conditional export via filtering processors - Processor Roles - Normalize and enrich - Filter and sample - Transform attributes for destination needs - Delivery Guarantees - Batch and retry behavior - Backpressure handling - Failure isolation per exporter - Validation - Check attribute presence - Verify counts per destination - Spot-check trace-log correlation fields

Step 1: Separate Pipelines by Signal, Then by Destination

The cleanest structure is one pipeline per signal type, because metrics, logs, and traces often need different sampling and different attribute requirements. Within a signal pipeline, you can still split delivery by destination using filtering processors.

Example intent:

• Traces pipeline exports to trace-prod for all services.
• Traces pipeline also exports to trace-compliance only for deployment.environment=prod.

Step 2: Use Filtering Processors for Conditional Export

Filtering is how you avoid sending everything to every exporter. A common pattern is:

1. Enrich attributes early so filters have stable inputs.
2. Apply a filter that keeps or drops data for a specific destination.
3. Export the remaining data to that destination.

A simple mental model: “If the filter passes, the exporter sees it.”

Step 3: Keep Attribute Contracts Consistent

Routing logic is only as good as the attributes it relies on. If deployment.environment is missing or inconsistent, routing becomes random. Enrichment processors should standardize values before any routing filters run.

For example, normalize service.namespace and service.name so that service.name is always the application name, not the library name. Then routing rules remain stable across deployments.

Step 4: Fan Out with Multiple Exporters, but Control the Blast Radius

Fan out can be done by listing multiple exporters in a single pipeline. That means every exporter receives the same stream. If one backend is slow or misconfigured, you want to avoid stalling the entire pipeline.

Operationally, you control this by:

• Using batching so transient slowness doesn’t immediately block ingestion.
• Ensuring exporters have independent retry behavior.
• Filtering so only the necessary subset reaches the secondary destination.

Example: Traces Routed by Environment and Fan Out to Two Backends

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  attributes/enrich:
    actions:
      - key: deployment.environment
        value: prod
        action: upsert
  filter/prod_only:
    error_mode: ignore
    traces:
      span:
        - attributes:
            - key: deployment.environment
              value: prod
              operator: strict

exporters:
  otlp/trace-prod:
    endpoint: trace-prod.example:4317
  otlp/trace-compliance:
    endpoint: trace-compliance.example:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/enrich]
      exporters: [otlp/trace-prod]

    traces/compliance:
      receivers: [otlp]
      processors: [attributes/enrich, filter/prod_only]
      exporters: [otlp/trace-compliance]

This uses two pipelines for the same signal: one always exports to the primary backend, and the other exports only when the filter passes. The result is fan out with controlled conditions.

Example: Metrics Split by Service Group

For metrics, you might route “payments” services to a backend with stricter retention, while sending everything else to the general backend.

A good rule is to group services by a stable attribute like service.name prefix or service.namespace, then filter based on that group. Avoid filtering on high-cardinality attributes like user.id or session.id, because it creates unpredictable routing and expensive processing.

Validation Checklist for Routing and Fan Out

• Confirm routing attributes exist before filters run.
• Compare exported counts per destination against expected service groups.
• Verify that trace-log correlation fields are present in both destinations when logs are also exported.
• Ensure secondary destinations receive only the intended subset, not the full stream.

When routing and fan out are designed this way, you get predictable delivery, less duplicated work, and fewer “why is this backend empty?” moments.

9.4 Batch Retry and Backpressure Handling in Pipelines

A collector pipeline has two jobs that often fight each other: move telemetry forward quickly, and avoid losing data when downstream systems slow down or fail. Batching helps with throughput, but it also changes failure behavior. When a batch export fails, you need a retry strategy that is predictable, bounded, and friendly to backpressure.

Batch Retry Fundamentals

Batching groups telemetry items into export requests. If an export attempt fails, the collector may retry the same batch. The key design choices are:

• Retryable vs non-retryable errors: network timeouts and temporary backend errors are usually retryable; malformed payloads are not.
• Retry limits: retries must stop, or you risk infinite queue growth.
• Backoff behavior: repeated immediate retries can amplify load on an already struggling backend.
• Idempotency expectations: retries can cause duplicates unless the backend or pipeline provides deduplication semantics.

A practical rule: treat retries as a way to survive transient issues, not as a substitute for correct data modeling.

Backpressure Mechanics

Backpressure is what happens when the pipeline produces faster than it can export. In a collector, backpressure typically shows up as:

• Queue growth between processors and exporters.
• Increased export latency because batches wait longer.
• Potential drops when queues hit capacity.

To handle this systematically, you need to decide what to protect:

• Protect memory by bounding queues.
• Protect latency by limiting batch size and wait time.
• Protect data quality by choosing which signal types can be dropped first.

Mind Map: Batch Retry and Backpressure

- Batch Retry and Backpressure Handling - Batch behavior - Batch size - Batch timeout - Export request granularity - Retry strategy - Retryable errors - timeouts - 5xx - connection resets - Non-retryable errors - schema violations - auth failures - Retry limits - max attempts - max elapsed time - Backoff - exponential - jitter - Backpressure controls - Bounded queues - memory limits - disk buffering if available - Flow control - block producers - shed load - Drop policy - oldest-first - signal priority - Operational checks - exporter errors - queue length - dropped item counters - end-to-end latency

Example: Reasoning Through a Failed Export

Imagine a pipeline exporting traces to a backend over OTLP. A batch of 2,000 spans is formed every 5 seconds or when it reaches a size threshold. The exporter sends the batch and receives a timeout.

A good retry strategy does the following:

1. Classifies the error as retryable because it looks like a transient network issue.
2. Retries with backoff so the backend has time to recover.
3. Caps retries so the same batch does not occupy resources forever.
4. Preserves ordering only where it matters. For telemetry, strict ordering is rarely required, but consistent correlation fields are.

If the backend later returns a 400-level error indicating invalid attributes, retries should stop. Retrying would just waste capacity while the queue grows.

Example: Backpressure with Bounded Queues

Suppose the exporter slows down due to backend throttling. The collector keeps accepting telemetry from instrumented services, so the internal queue grows. Once the queue hits its limit, the collector must choose between blocking ingestion and dropping.

A common, practical approach is:

• Prefer bounded blocking for a short period to absorb brief slowdowns.
• Drop with a clear policy when the queue remains full.

For example, you might drop the oldest batches first to keep the most recent telemetry. That choice is not perfect, but it prevents the system from spending all resources on stale data.

Example: Collector Configuration Pattern

Below is a conceptual pattern showing how batching, retry, and queue limits are typically expressed. Exact field names vary by collector distribution and version, so treat this as a design template.

exporters:
  otlp:
    endpoint: https://backend.example/v1/otlp
    retry_on_failure:
      enabled: true
      max_elapsed_time: 30s
      initial_interval: 500ms
      max_interval: 5s
    sending_queue:
      enabled: true
      queue_size: 10000
      num_consumers: 2
processors:
  batch:
    timeout: 5s
    send_batch_size: 2000

If you see queue length climbing while exporter errors are retryable, you likely need either a larger queue (if memory allows) or a smaller batch/timeout to reduce per-request work. If exporter errors are non-retryable, the fix is upstream data correctness or credentials, not more retries.

Operational Checks That Prevent Surprises

When batch retry and backpressure are working, you should observe:

• Exporter error counters rising only briefly during transient incidents.
• Queue length returning to baseline after recovery.
• Dropped item counters staying near zero under normal load.
• Stable end-to-end latency without sawtooth patterns that indicate repeated timeouts.

A simple mental model helps: batching increases efficiency, retries increase resilience, and backpressure controls prevent resource exhaustion. The collector is happiest when those three are tuned together rather than independently.

9.5 Example Collector Config for a Multi Service Deployment

A multi-service deployment usually means you want three things at once: consistent identity across services, signal-specific processing, and predictable routing to one or more backends. The OpenTelemetry Collector is a good fit because it separates ingestion (receivers), transformation (processors), and delivery (exporters) into explicit pipeline stages.

Foundational Layout for Multi Service Pipelines

Start by deciding how telemetry enters the collector. A common pattern is a single OTLP endpoint that accepts data from many services, then routes by signal type and service identity.

In practice, you’ll typically:

• Use one OTLP receiver for all services.
• Split pipelines by signal type: traces, metrics, and logs.
• Normalize resource attributes so every service has the same identity fields.
• Apply light filtering early to reduce noise and cost.
• Batch and retry to smooth out backend hiccups.

Mind Map: Multi Service Collector Responsibilities

- Collector - Receivers - OTLP - gRPC endpoint - HTTP endpoint - auth and TLS - Processors - Resource normalization - service.name - service.namespace - service.instance.id - Attribute transforms - rename keys - add environment - Filtering - drop low-value spans - drop debug logs - Batching - batch size - flush interval - Pipelines - Traces pipeline - span enrichment - sampling or filtering - export - Metrics pipeline - metric relabeling - aggregation adjustments - export - Logs pipeline - log level mapping - correlation fields - export - Exporters - Backend A - traces - metrics - logs - Backend B - optional routing - Observability of the collector - collector metrics - internal logs

Example Collector Configuration

Below is a compact but realistic configuration skeleton. It assumes:

• Services send OTLP to the collector.
• You want consistent resource attributes.
• You export all three signals to the same backend.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  resource:
    attributes:
      - key: service.namespace
        value: "payments"
        action: upsert
      - key: deployment.environment
        value: "prod"
        action: upsert

  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(attributes["otel.log.severity_text"], attributes["severity_text"]) where attributes["severity_text"] != nil

  batch:
    send_batch_size: 1024
    timeout: 5s

exporters:
  otlp:
    endpoint: backend.example.internal:4317
    tls:
      insecure: false

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [resource, transform, batch]
      exporters: [otlp]

This example shows the core idea: one receiver, three pipelines, and processors reused where they make sense. The transform processor is placed only in the logs pipeline because it’s log-specific.

Attribute Normalization That Actually Helps

Resource normalization is where multi-service setups either stay manageable or turn into a guessing game. If one service uses service.name and another uses a custom key, queries become inconsistent.

A practical approach is to:

• Ensure service.name is provided by instrumentation.
• Upsert service.namespace and deployment.environment at the collector when they’re missing.
• Keep the collector’s normalization rules simple and deterministic.

In the example, service.namespace is set to payments and deployment.environment to prod. In a real deployment, you might set these based on the incoming network segment or a known deployment label, but the key point is that the collector should enforce a consistent schema.

Routing by Service Identity Without Duplicating Pipelines

If you need different backends per service, you can route using processors that filter or set attributes, then export from signal pipelines to different exporters. The simplest method is to keep one pipeline per signal and use conditional processors.

Here’s a small pattern that drops noisy logs from a specific service while keeping everything else:

processors:
  filter/logs:
    error_mode: ignore
    logs:
      exclude:
        match_type: strict
        attributes:
          - key: service.name
            value: "debug-worker"

Then add filter/logs to the logs pipeline’s processor list before batch. This keeps the configuration readable because the routing logic stays close to the signal it affects.

Operational Checks for a Multi Service Collector

After deploying, verify three things in order:

1. The collector receives data from multiple services on the same OTLP endpoint.
2. Resource attributes are present and consistent across services.
3. Each pipeline exports the expected signal type.

A quick sanity check is to look for a single service’s traces, metrics, and logs sharing the same service.name, service.namespace, and deployment.environment. If those fields don’t line up, the issue is usually instrumentation or resource normalization, not the backend.

Mind Map: Configuration Flow

processors:
  filter/health:
    traces:
      span:
        - attributes:
            - key: http.route
              value: /health
            - key: http.status_code
              value: 200
          action: drop
  attributes/prune:
    actions:
      - key: http.request.body
        action: delete
      - key: user.id
        action: delete
      - key: url.full
        action: delete

receivers:
  otlp:
    protocols:
      grpc:

processors:
  resource/detect:
    detectors: [env, k8s]
  resource/enrich:
    attributes:
      - key: service.namespace
        from_attribute: k8s.namespace.name
        action: upsert
      - key: deployment.environment
        from_attribute: env.DEPLOY_ENV
        action: upsert

exporters:
  otlp:
    endpoint: https://backend.example

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource/detect, resource/enrich]
      exporters: [otlp]

processors:
  metricstransform:
    transforms:
      - include: http.server.request.duration
        match_type: strict
        action: update
        operations:
          - action: scale
            from: 0.001
          - action: update_label
            label: http.route
            new_label: http.route

processors:
  transform/logs:
    log_statements:
      - context: log
        statements:
          - set(attributes["trace_id"], attributes["traceId"]) where attributes["trace_id"] == nil and attributes["traceId"] != nil
          - set(attributes["span_id"], attributes["spanId"]) where attributes["span_id"] == nil and attributes["spanId"] != nil
          - set(attributes["trace_id"], lower(attributes["trace_id"])) where attributes["trace_id"] != nil
          - set(attributes["span_id"], lower(attributes["span_id"])) where attributes["span_id"] != nil

exporters:
  otlphttp:
    endpoint: https://telemetry.example.com/v1/otlp
    headers:
      Authorization: "Bearer ${TELEMETRY_TOKEN}"

processors:
  attributes:
    actions:
      - key: http.user_agent
        action: delete
      - key: user.id
        action: delete

exporters:
  otlphttp:
    endpoint: https://telemetry.example.com/v1/otlp
    headers:
      Authorization: "Bearer ${TELEMETRY_TOKEN}"

processors:
  resource:
    attributes:
      - key: service.namespace
        value: payments
        action: upsert
      - key: deployment.environment
        value: production
        action: upsert

for each telemetry_item:
  ensure resource identity keys exist
  ensure operation keys exist for detected interaction type
  ensure outcome fields match error conditions
  if missing required keys:
    either drop item or add validation_error attribute
  if attribute types mismatch:
    normalize type or drop offending attribute

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch: {}

exporters:
  otlphttp:
    endpoint: https://backend.example/otlp
    tls:
      insecure: false

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  batch: {}
  resource: { attributes: [] }
exporters:
  otlphttp:
    endpoint: http://backend:8080
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlphttp]

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  filter/low_value:
    traces:
      span_attributes:
        - key: "http.route"
          values: ["/health"]
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 64

exporters:
  otlphttp:
    endpoint: https://backend.example/v1/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/low_value, memory_limiter, batch]
      exporters: [otlphttp]