Attributing LLM Costs: A Guide to OpenTelemetry and TraceQL

Why is LLM cost attribution a critical engineering challenge?

Most engineering organizations already collect telemetry for their large language model integrations. Spans are generated, token counts are recorded, and traces are shipped to observability platforms. However, a significant gap remains in the data pipeline. The missing dimension is the one that finance departments and engineering leads actually require: ownership of the spend. When a cloud invoice arrives, it typically presents a single line item per API key. If that key is shared across multiple services, the resulting number is useless for chargeback purposes.

Without granular tagging, it is impossible to distinguish the spending of a platform team from that of a customer support bot. The design goal for a robust attribution system is narrow but essential. Every large language model call must carry enough labels to group spend by team, by project within that team, and by feature inside that project. This three-level hierarchy provides sufficient detail without overwhelming the dashboard with noise. Standardizing the pipeline on OpenTelemetry and OpenInference ensures that the labels, wire format, and storage are swappable, allowing for portability across different backend tools.

The payoff for implementing this system is immediate and tangible. In one documented case, a single team's monthly spend quietly increased from a few hundred dollars to over a thousand due to a persistent retry loop. The org-level dashboard never flinched because the total spend was diluted across the entire organization. However, the per-team view caught the anomaly in a single day. This visibility allows engineering leaders to identify inefficiencies and correct them before they become significant budgetary issues.

What attributes must be tagged at the gateway level?

Tagging should occur at the gateway, not within each individual service. Running an LLM gateway ensures that every call to every provider passes through a single point of control. This architecture allows the gateway to see the model, token counts, and request context together. A new service gets attribution for free as long as it routes through the gateway and forwards the necessary context headers. This centralization simplifies instrumentation and reduces the risk of inconsistent tagging across different microservices.

The attributes required for cost calculation come directly from OpenInference semantic conventions. These include the model name, the prompt token count, and the completion token count. The attribution attributes are custom but standardized. They include the team identifier, the project identifier, and the feature identifier. Cost itself is not stored as a span attribute. Instead, it is computed at query time from token counts and a price lookup table. This approach is critical because prices change frequently. Freezing rates at write time would result in inaccurate historical reporting.

Some attributes that seem useful should be dropped to reduce complexity and privacy risks. User identifiers, for example, create a significant privacy liability. Putting a user identifier on every span means every trace becomes personally identifiable information. This inheritance of compliance obligations for retention, access, and deletion rarely justifies the attribution win. Similarly, request identifiers are pure redundancy. A trace already has a trace ID, and every span has a span ID. Anywhere a request ID was thought to be needed, the existing trace ID was already sufficient.

How does the OpenTelemetry collector handle missing data?

The OpenTelemetry collector configuration is the backbone of this attribution strategy. The pipeline receives traces via OTLP, processes them, and exports them to a backend like Grafana Tempo. One processor worth highlighting is the transform processor. It is used to backfill team identifiers with a sentinel value when a service forgets to include the header. This ensures that unlabeled spend shows up as unattributed rather than vanishing from the dashboard entirely.

Cost with no label is cost that you will never find. The transform processor must be placed before the batch processor. This ordering ensures that the backfill happens per span while the data is still cheap to touch. If the batch processor runs first, the data is grouped, and individual span attributes are harder to modify efficiently. Keeping the price table out of the collector is another crucial lesson. Encoding per-model rates as collector attributes once led to configuration drift. Every price change required a config deploy, and the rates quickly fell out of sync with actual billing.

The exporter configuration sends the processed traces to the chosen backend. For many organizations, this is Grafana Tempo, which supports TraceQL for powerful aggregation queries. The service configuration ties the receivers, processors, and exporters together. This modular design allows teams to swap out components without rewriting the instrumentation code. The use of open semantic conventions ensures that the labels are not tied to a specific vendor. This reversibility is a key feature of the architecture, protecting the organization from vendor lock-in.

How do you roll up costs using TraceQL in Grafana?

Grafana Tempo stores spans, not dollars. Therefore, the rollup process involves two distinct steps. First, TraceQL pulls token sums grouped by the attribution attributes. Second, a small price map turns those tokens into cost downstream. The initial query aggregates output token counts, which is the primary metric to watch. Completion tokens are usually where the money and the runaways are. The query filters for specific teams and selects the relevant project, feature, model, and token counts.

Grouping by the model name is essential. A mini-tier model and a frontier model can differ by more than an order of magnitude per token. Summing raw tokens across models hides whether a team is expensive because of volume or because of model choice. The dollar step is deliberately kept simple. It involves a lookup from the model name to input and output prices per thousand tokens. Multiplying and summing per team allows for re-pricing history when a provider changes rates. Keeping the logic external to the storage layer ensures that historical data remains accurate regardless of future pricing changes.

Dropping the team filter and grouping by it instead creates an all-teams board. This view provides a high-level overview of spending across the organization. It allows leaders to identify which teams are driving the most cost and which projects are the most resource-intensive. The granularity of the attribution attributes ensures that the data is actionable. Teams can drill down into specific features to understand the drivers of their spend. This level of detail is necessary for effective chargeback and budget management.

What alerting strategy prevents runaway costs?

Cost attribution is reporting, but the thing that earns its keep is the alert. The rule run is week-over-week on output tokens per team. If the completion token total for any team is more than double the same window last week, an alert is triggered. Output tokens are chosen over input tokens because runaway failure modes, such as retry storms, looping agents, or prompt-chaining bugs, show up as generation volume first.

Using a relative jump, such as a two-fold increase, normalizes across team size. A fixed dollar ceiling would either page constantly for large teams or never fire for small ones. The team whose spend doubled in the story above would have tripped a two-fold rule on day one. It did not trip a dollar alert because the absolute number was still small against the org total. Small against the org, but doubled for the team, is exactly the blind spot per-team attribution exists to close.

The alert should be routed to whoever owns the team's budget, not a shared channel where it gets ignored. This direct routing ensures that the right people are notified and can take immediate action. For steady traffic, a two-fold increase is a reliable indicator of a problem. For genuinely spiky workloads, the ratio can be raised or the comparison window widened. The bias is toward a slightly noisy page over a silent doubling, as catching issues early is more valuable than minimizing alert fatigue.

What are the open questions in LLM cost attribution?

Caching breaks the token-to-cost math. Cached prompt tokens bill at a different rate, sometimes for free. Tagging cache hits cleanly enough to price them right remains a challenge. Streaming and cancelled generations also pose questions. If a client disconnects mid-stream, what is the honest output token count, and does the provider bill for tokens generated after the cancel? These edge cases require careful handling to ensure accurate attribution.

Feature-level granularity has a ceiling. There is a desire for per-prompt-version attribution, but every level deeper adds one more label that nobody reads. The balance between detail and usability is a constant trade-off. Whether the two-fold week-over-week alert should be per-team is another open question. Some teams are spiky by nature, and one global ratio serves both steady and spiky teams imperfectly. If you have wired cached-token pricing into a span-based cost model in a way that survives provider changes, the engineering community is eager to hear how.

Related Reading

For more insights on securing the tools used in this pipeline, see Developer Endpoint Protection: Securing the Modern Workstation. Additionally, understanding the broader context of enterprise AI infrastructure can be found in SpaceX Acquisition of Cursor Reshapes Enterprise AI Infrastructure.