After completing this chapter you will be able to:

• Define and instrument the core latency metrics that govern user experience in LLM systems: TTFT, TPOT/ITL, end-to-end latency, and p50/p95/p99 percentiles.
• Design an observability stack using OpenTelemetry that captures traces, metrics, logs, token usage, and cost attribution for every request.
• Implement semantic caching, prompt caching, and model routing to cut inference cost and latency without sacrificing quality.
• Detect embedding drift and data drift in production RAG and agent pipelines using statistical and semantic methods.
• Apply architectural patterns (batching, quantization, KV-cache sharing, disaggregated prefill/decode, autoscaling) that keep high-concurrency LLM services reliable and cost-effective.
• Build dashboards, alerts, and SLOs that experienced teams at scale actually use.
• Avoid the most common production failure modes that turn promising prototypes into expensive, slow, or unreliable systems.

Prerequisites

Who this is for

• Expats, students, career changers, junior professionals, and self-learners who already know the basics of building an LLM app (calling an API, simple RAG, or an agent loop) and now need to run it in production.
• Engineers who have shipped a prototype and are suddenly facing latency spikes, unpredictable bills, or silent quality degradation.

Required knowledge

• Comfortable reading Python (or equivalent) and REST/gRPC concepts.
• Basic understanding of tokens, embeddings, and what an LLM inference call does.
• Familiarity with containers, HTTP, and simple monitoring (logs or Prometheus-level metrics) is helpful but not required; every concept is introduced from first principles.

Estimated reading time

90–120 minutes for a careful first pass. Plan additional time for the exercises and code experiments.

Difficulty

Intermediate. The material starts from zero on observability and cost concepts, then moves into production architecture. No prior SRE or MLOps experience is assumed.

Why This Matters

LLM applications behave differently from classic web services. A single user request can trigger multiple model calls, retrieval steps, tool invocations, and guardrails. Latency compounds. Cost is metered per token. Quality can silently drift when the distribution of user prompts changes. In high-concurrency environments these effects become first-order product and business risks. community.ibm

Industry teams treat LLM serving with the same rigor Site Reliability Engineering applies to any critical service: they define SLIs (latency, availability, output quality), set SLOs, implement circuit breakers and fallbacks, and pair autoscaling with queue-based load regulation. Observability for LLMs therefore includes p95/p99 latency, token counts per request, cost per call, cache hit rates, and semantic drift monitoring. medium

Companies running production inference at scale—whether using managed APIs or self-hosted stacks such as vLLM—rely on these practices to keep unit economics viable and user experience predictable. Without them, teams discover too late that GPU spend has ballooned, p99 latency has made the product unusable, or embeddings have drifted so far that retrieval quality collapsed. This chapter gives you the mental models, metrics, and patterns that turn those risks into controllable engineering problems.

First Principles

What “observability” means for an LLM application

Observability is the ability to understand the internal state of a system from its external outputs—logs, metrics, and traces. For a classic HTTP API you care about request rate, error rate, and latency. For an LLM application you must also answer:

• How many input and output tokens did this request consume, and what did that cost?
• How long did the user wait before the first token appeared (TTFT)?
• How fast did subsequent tokens arrive (TPOT / ITL)?
• Which stage of the pipeline (retrieval, prompt construction, model inference, tool call, guardrail) dominated latency or cost?
• Has the distribution of incoming prompts or embeddings shifted so that quality is degrading even though the service is “up”?

If you cannot answer these questions per request, per feature, and per model, you do not yet have production observability. 21medien

Why LLM latency is multi-modal

Classic services often report a single latency number. LLM inference has distinct phases with different physics:

1. Queue / network time — waiting for a free worker or crossing the network.
2. Prefill — the model processes the entire prompt and builds the key-value (KV) cache. This dominates Time to First Token.
3. Decode — the model generates one token at a time, each step attending over the growing context. This determines Time per Output Token.

The clean decomposition used across the industry is:

TTFT and TPOT move independently. Optimizing the wrong one wastes effort and money. docs.anyscale

Why cost is a first-class metric

Every token has a price. Input tokens, output tokens, and (when available) cached tokens are billed differently. A single verbose system prompt repeated on every request, or an agent that loops unnecessarily, can multiply cost by 5–10× compared with a carefully engineered path. Cost must therefore be attributed per request, per feature, and per model—exactly like latency. maviklabs

Why drift matters even when the model is frozen

You may freeze the model weights, yet the world changes. User language evolves, product catalogs update, regulations shift. The embeddings of new prompts drift relative to the baseline you used to tune retrieval or routing. Statistical and semantic drift detection tells you when the ground has moved under your feet so you can retrain, re-index, or adjust thresholds before quality collapses. docs.aws.amazon

With these principles in place we can define the concrete metrics and architecture.

Core Concepts

Latency Metrics That Actually Matter

Time to First Token (TTFT)

Explanation. TTFT is the elapsed time from the moment the client sends the request until the first output token is received. It includes network transit, any server-side queueing, and the entire prefill phase. clickhouse

Intuition. In a streaming chat UI, TTFT is the awkward silence before the model “starts talking.” Users notice it immediately.

Real-world example. A support chatbot with a 4 k-token system prompt plus retrieved documents may show TTFT of several seconds on a cold or heavily loaded GPU, even if subsequent tokens stream quickly.

Visual explanation.

Common misconception. “Faster GPUs always fix TTFT.” Prefill scales with prompt length and batch size; queueing and cold starts often dominate. You must measure the components separately. tianpan

Time per Output Token (TPOT) / Inter-Token Latency (ITL)

Explanation. TPOT is the average time between consecutive output tokens after the first. For a single request:

ITL usually refers to the individual gaps; TPOT is their average (request-weighted). handbook.modular

Intuition. TPOT is the “reading speed” of the response. Even with excellent TTFT, a high TPOT makes the stream feel sticky.

Real-world example. Interactive coding assistants target roughly 20–100 tokens/s per request (TPOT ≈ 10–50 ms) so the stream feels faster than a human can read. clickhouse

Common misconception. “Throughput (tokens/s server-wide) is the same as user-perceived speed.” Server throughput can be high while individual requests suffer from large batching delays. Always report per-request TPOT percentiles. digitalocean

End-to-End Latency and Percentiles

Explanation. End-to-end (E2E) latency is wall-clock time from request sent to final token received. Always track p50, p95, and p99. A healthy p50 with a large p99 gap signals tail latency caused by queueing, stragglers, or occasional long prompts. 21medien

When to use which metric

tp88

Throughput, Concurrency, and Goodput

Throughput is usually measured as tokens generated per second across the whole serving fleet, or requests per second. Concurrency is the number of in-flight requests. Because decode is memory-bandwidth bound and prefill is compute bound, raising concurrency past a point increases throughput but also raises TTFT and TPOT for every request—the classic latency-throughput tradeoff. medium

Goodput is the fraction of requests that meet your latency SLOs. Maximizing raw tokens/s while goodput collapses is a common production mistake. Watch the saturation curve: as load rises, TTFT and TPOT percentiles climb; goodput eventually falls. Capacity planning uses that curve. learnaivisually

Token Usage and Cost Attribution

Every successful observability pipeline records, per request:

• Input tokens
• Output tokens
• Cached tokens (when the provider supports prompt caching)
• Model identifier and provider
• Feature / route / tenant tags
• Estimated cost (using the provider’s current price table)

Counters and histograms of these values feed cost dashboards and budget alerts. Without per-feature attribution you cannot answer “which product surface is burning the budget?” jobsbyculture

Semantic Caching vs Exact / Prompt Caching

Exact caching returns a stored response only when the prompt string matches exactly. Natural language almost never repeats verbatim, so hit rates stay low.

Prompt caching (offered by several major providers) caches the computation of a long, stable prefix (system prompt + tools + few-shot examples). Subsequent requests that share the same prefix pay a reduced rate for those tokens. Typical savings on agent and chat workloads: 60–80% of the prefix cost. aisuperior

Semantic caching embeds the incoming query, compares it (usually by cosine similarity) to embeddings of previously answered queries, and, if similarity exceeds a threshold, returns the cached answer without calling the LLM. Hit rates of 30–60% (and higher on repetitive workloads) are common when the threshold is tuned carefully. dev

Intuition. Exact cache = “same string.” Prompt cache = “same expensive prefix.” Semantic cache = “same meaning.”

Common misconception. “Semantic cache is always safe.” If the threshold is too low you serve wrong answers. Start around 0.8 cosine similarity, raise it for precision-sensitive domains, and always monitor quality alongside hit rate. dev

Embedding Drift and Data Drift

Data drift is a change in the distribution of inputs. Embedding drift is the same phenomenon observed in the high-dimensional vector space produced by your embedding model.

Detection workflow used in production:

1. Establish a baseline distribution of prompt (or document) embeddings from a stable period.
2. Continuously sample production embeddings.
3. Compare distributions with a metric suited to high dimensions (Wasserstein distance, centroid shift, classifier AUC, etc.).
4. Alert when the distance exceeds a threshold; then sample the drifted prompts and optionally use an LLM-as-judge for semantic classification of the change. apxml

Without this loop, retrieval quality and routing decisions degrade silently.

Observability Pillars Applied to LLMs

OpenTelemetry has become the vendor-neutral standard for this instrumentation. medium

Deep Dive

Anatomy of a Production LLM Request Path

A realistic high-concurrency path looks like this:

Every box must emit metrics and spans. The gateway is also the natural place for fallbacks, load balancing across providers, and cost budgets. truefoundry

Prefill vs Decode and Disaggregation

Prefill is compute-heavy and benefits from high FLOPS; decode is memory-bandwidth heavy and benefits from high HBM bandwidth and efficient KV-cache management. Modern serving systems (vLLM and derivatives, production stacks with LMCache, etc.) therefore:

• Use PagedAttention / continuous batching so that decode steps from many requests share the GPU efficiently.
• Optionally disaggregate prefill and decode onto different pools so that long-prefill jobs do not stall interactive decode traffic.
• Share or offload KV cache across instances (prefix-aware routing) to avoid recomputing common prefixes. youtube

These techniques simultaneously improve TTFT, TPOT, and cost per token.

Quantization, Batching, and Model Tiering

community.ibm

Cost Optimization Playbook (2026)

Experienced teams report that most production bills can be cut 47–80% (sometimes more) without quality loss by applying a short list of high-leverage moves in order: jobsbyculture

1. Meter everything first. Per-request cost telemetry tagged by feature and model. Build a “top-10 routes by cost” dashboard.
2. Enable prompt caching on every route with a stable prefix longer than ~500–1 k tokens. Savings of 60–90% on the prefix are common.
3. Route by difficulty. Send the 60–80% of easy requests to a small/cheap model; escalate only when a quality or confidence signal fails. Savings 50–70% on mixed workloads.
4. Batch APIs for any work that is not user-facing and real-time (often ~50% discount).
5. Shape outputs. Strict structured outputs, explicit max tokens, and removal of unnecessary verbosity cut output tokens 20–40%.
6. Semantic caching for repetitive natural-language queries (30–60%+ when tuned).
7. Fine-tune or distill a small specialist model only when volume justifies the MLOps cost (roughly >100 M tokens/month on a stable task).

Never optimize before you can attribute cost. The first dashboard often reveals a single verbose agent loop or an uncached system prompt that dominates spend.

Drift Detection Internals

For embedding drift the following statistics are practical:

• Centroid shift: $\|\mu_{\text{ref}} - \mu_{\text{current}}\|_2$
• Wasserstein / Earth-Mover distance on the embedding distributions (or on PCA-reduced vectors)
• Classifier two-sample test: train a binary classifier to distinguish baseline vs current embeddings; AUC near 0.5 means no drift, high AUC means separable (drifted) distributions. apxml

When an alert fires, sample the new prompts and run a lightweight LLM-as-judge prompt that classifies the semantic nature of the change (new product names, language shift, adversarial inputs, etc.). That classification drives the remediation playbook (re-index, retrain router, tighten guardrails). docs.aws.amazon

SLIs, SLOs, and Alerting

Define SLIs that map directly to user pain:

• Availability: fraction of requests that return a non-error completion within the timeout.
• Latency: p95 TTFT < X ms, p95 TPOT < Y ms, p99 E2E < Z ms.
• Quality: fraction of sampled outputs that pass an automatic judge or user thumbs-up.
• Cost: cost per successful request or per active user, with budget burn-rate alerts.

Pair each SLO with a multi-window, multi-burn-rate alert (the same pattern used by Google SRE) so that you are paged for fast burns and ticketed for slow burns. Include circuit breakers that shed load or fall back to a cheaper/faster model when error budgets are exhausted. community.ibm

Historical Context (condensed)

Scaling laws (Kaplan et al., 2020) showed that loss improves predictably with model size, data, and compute, driving the race to ever-larger models. Once those models reached production, the bottleneck shifted from training to inference economics and reliability. Systems papers and open-source engines (vLLM, TensorRT-LLM, production stacks with KV-cache sharing) then focused on continuous batching, paged attention, and disaggregation. Observability standards (OpenTelemetry GenAI semantic conventions) and cost-control patterns (prompt caching, semantic caches, intelligent routers) matured between 2024 and 2026 into the toolkit described in this chapter. arxiv

Practical Examples

Beginner — Instrumenting a Single Chat Completion

Goal: record TTFT, TPOT, token counts, and estimated cost for one OpenAI-compatible call.

import time
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# Simple price table (update with real 2026 prices)
PRICE_PER_1K = {
    "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
}

def chat_with_metrics(messages, model="gpt-4o-mini"):
    t0 = time.perf_counter()
    first_token_time = None
    chunks = []
    stream = client.chat.completions.create(
        model=model,
        messages=messages,
        stream=True,
        stream_options={"include_usage": True},
    )
    usage = None
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.perf_counter()
            chunks.append(chunk.choices[0].delta.content)
        if getattr(chunk, "usage", None):
            usage = chunk.usage
    t_end = time.perf_counter()

    text = "".join(chunks)
    ttft = (first_token_time - t0) if first_token_time else None
    e2e = t_end - t0
    # Approximate output tokens if usage missing
    out_tokens = usage.completion_tokens if usage else max(len(text.split()), 1)
    tpot = (e2e - ttft) / max(out_tokens - 1, 1) if ttft else None

    in_tokens = usage.prompt_tokens if usage else 0
    cost = (
        in_tokens / 1000 * PRICE_PER_1K[model]["input"]
        + out_tokens / 1000 * PRICE_PER_1K[model]["output"]
    )

    return {
        "text": text,
        "ttft_s": ttft,
        "tpot_s": tpot,
        "e2e_s": e2e,
        "input_tokens": in_tokens,
        "output_tokens": out_tokens,
        "cost_usd": cost,
    }

# Example call
result = chat_with_metrics([
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "Explain TTFT in one sentence."},
])
print(result)

What you learn. Even a few lines of instrumentation surface the two latency numbers and the dollar cost. In production you would emit these as OpenTelemetry metrics/spans instead of printing.

Intermediate — Semantic Cache in Front of the Model

import hashlib
import numpy as np
from dataclasses import dataclass
from typing import Optional, List, Tuple

# Assume embed(text) -> np.ndarray and llm_complete(prompt) -> str
# and a vector store with search(vector, k) -> List[(id, score, payload)]

SIMILARITY_THRESHOLD = 0.85
TTL_SECONDS = 3600

@dataclass
class CacheEntry:
    embedding: np.ndarray
    response: str
    created_at: float

class SemanticCache:
    def __init__(self, embed_fn, store, llm_fn, threshold=SIMILARITY_THRESHOLD):
        self.embed = embed_fn
        self.store = store
        self.llm = llm_fn
        self.threshold = threshold

    def get_or_generate(self, query: str, now: float) -> Tuple[str, bool]:
        q_emb = self.embed(query)
        hits = self.store.search(q_emb, k=1)
        if hits:
            _id, score, payload = hits[0]
            if score >= self.threshold and now - payload["created_at"] < TTL_SECONDS:
                return payload["response"], True  # cache hit
        # miss
        response = self.llm(query)
        self.store.upsert(
            id=hashlib.sha256(query.encode()).hexdigest(),
            vector=q_emb,
            payload={"response": response, "created_at": now, "query": query},
        )
        return response, False

Operational notes.

• Start with threshold ≈ 0.8–0.85; raise for legal/medical domains. dev
• Record hit/miss as a metric and sample cached answers for quality audits.
• Combine with exact-match cache on normalized queries for free additional hits.
• Set TTLs according to content volatility (minutes for news, hours/days for stable docs).

Advanced — OpenTelemetry Trace for a RAG + LLM Pipeline

Conceptual span hierarchy (Python OpenTelemetry SDK style):

root: user_request
├── retrieval
│   ├── embed_query
│   └── vector_search
├── prompt_assembly
├── llm.chat (gen_ai.system=openai, gen_ai.request.model=..., 
│            gen_ai.usage.input_tokens=..., gen_ai.usage.output_tokens=...)
│   └── (events: first_token, etc.)
├── guardrail_check
└── cost_attribution (attributes: cost_usd, feature=support_bot)

Emit histograms for llm.ttft, llm.tpot, llm.e2e and counters for tokens and cost. Export to a collector that scrapes Prometheus-compatible metrics and forwards traces to your backend (Jaeger, Grafana Tempo, vendor APM, etc.). freecodecamp

Use the OpenTelemetry GenAI semantic conventions so that any compliant backend can display model name, token counts, and latency without custom parsing. mlflow

Code Examples

Production-oriented metric helpers

from opentelemetry import metrics, trace
from opentelemetry.trace import Status, StatusCode
import time

meter = metrics.get_meter("llm.observability")
tracer = trace.get_tracer("llm.observability")

ttft_hist = meter.create_histogram("llm.ttft", unit="s")
tpot_hist = meter.create_histogram("llm.tpot", unit="s")
e2e_hist = meter.create_histogram("llm.e2e", unit="s")
token_counter = meter.create_counter("llm.tokens")
cost_counter = meter.create_counter("llm.cost", unit="USD")

def instrumented_chat(client, model, messages, feature: str, price_table: dict):
    with tracer.start_as_current_span("llm.chat") as span:
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("feature", feature)

        t0 = time.perf_counter()
        first = None
        text_parts = []
        usage = None

        try:
            stream = client.chat.completions.create(
                model=model,
                messages=messages,
                stream=True,
                stream_options={"include_usage": True},
            )
            for chunk in stream:
                if chunk.choices and chunk.choices[0].delta.content:
                    if first is None:
                        first = time.perf_counter()
                        span.add_event("first_token")
                    text_parts.append(chunk.choices[0].delta.content)
                if getattr(chunk, "usage", None):
                    usage = chunk.usage
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR))
            raise

        t1 = time.perf_counter()
        ttft = (first - t0) if first else None
        e2e = t1 - t0
        out_tok = usage.completion_tokens if usage else 0
        in_tok = usage.prompt_tokens if usage else 0
        tpot = (e2e - ttft) / max(out_tok - 1, 1) if ttft and out_tok > 1 else None

        attrs = {"model": model, "feature": feature}
        if ttft is not None:
            ttft_hist.record(ttft, attrs)
        if tpot is not None:
            tpot_hist.record(tpot, attrs)
        e2e_hist.record(e2e, attrs)
        token_counter.add(in_tok, {**attrs, "direction": "input"})
        token_counter.add(out_tok, {**attrs, "direction": "output"})

        cost = (
            in_tok / 1000 * price_table[model]["input"]
            + out_tok / 1000 * price_table[model]["output"]
        )
        cost_counter.add(cost, attrs)
        span.set_attribute("gen_ai.usage.input_tokens", in_tok)
        span.set_attribute("gen_ai.usage.output_tokens", out_tok)
        span.set_attribute("cost_usd", cost)

        return "".join(text_parts)

This pattern is the minimum viable production instrumentation: one function wraps every LLM call, records the four critical signals, and stays vendor-neutral via OpenTelemetry. medium

Simple difficulty router (cost control)

def route_and_complete(query: str, small_client, large_client, judge_fn):
    """
    Try cheap model first; escalate only if judge says quality is insufficient.
    """
    draft = small_client.complete(query)
    if judge_fn(query, draft) >= 0.85:  # confidence or rubric score
        return draft, "small"
    final = large_client.complete(query)
    return final, "large"

Pair with an eval set of a few hundred examples so you know the small model’s pass rate before you turn the router on in production. jobsbyculture

Diagrams

Decision tree: which latency knob to turn

Cost-control control loop

Tables

Latency metrics cheat comparison

docs.nvidia

Caching strategies

youtube

Cost levers ranked by typical impact

jobsbyculture

Common Mistakes

1. Tracking only average latency
Why it happens: averages hide tails.
How to avoid: always export p50/p95/p99 for TTFT, TPOT, and E2E. Alert on the percentiles that appear in your SLO. tianpan

2. Optimizing tokens/s while goodput collapses
Why it happens: bigger batches look great on GPU utilization dashboards.
How to avoid: plot the saturation curve; define goodput as “requests meeting TTFT/TPOT SLOs” and maximize that. learnaivisually

3. No cost attribution
Why it happens: the bill arrives as one line item from the provider.
How to avoid: tag every call with feature/tenant/model and maintain a price table; build a top-N cost dashboard before any other optimization. jobsbyculture

4. Semantic cache with an untested threshold
Why it happens: teams chase hit rate.
How to avoid: start conservative (≈0.85), measure answer correctness on a held-out set, and monitor cached vs fresh quality in production. dev

5. Ignoring embedding drift
Why it happens: the model weights are frozen, so “nothing can change.”
How to avoid: baseline embeddings, compute a high-dimensional drift statistic on a schedule, and alert + sample when it moves. docs.aws.amazon

6. One giant model for every request
Why it happens: simplest integration.
How to avoid: classify difficulty or use a cheap model first with confidence-based escalation. maviklabs

7. Logging full prompts and completions without redaction
Why it happens: debugging convenience.
How to avoid: scrub PII and secrets in the instrumentation layer before export; sample rather than store 100% of traffic when volumes are high. mlflow

8. No fallback or circuit breaker
Why it happens: happy-path demos.
How to avoid: timeouts, retries with jitter, fallback to a smaller model or cached answer, and load shedding when error budgets burn. community.ibm

Real Industry Examples

SRE-style LLM serving. Production guidance for scaling LLMs explicitly re-uses classic SRE practices: SLIs/SLOs for latency and quality, circuit breakers, queue-based load regulation, and monitoring of the full path including vector databases and embedding services. community.ibm

vLLM Production Stack. The open-source vLLM ecosystem has been extended with KV-cache sharing (LMCache), prefix-aware routing, and first-class observability so organizations can run multi-instance deployments on Kubernetes with higher hit rates on shared prefixes and clearer autoscaling signals. youtube

Enterprise observability stacks. Teams instrument every LLM call with OpenTelemetry, export to Prometheus/Grafana (or equivalent), and track exactly the metrics listed in this chapter—p95/p99 latency, token counts, cost per feature, cache hit rates, and queue depth—then add continuous evaluation on a sample of traffic. 21medien

Cost reduction case patterns (2025–2026 literature). Multiple independent engineering write-ups report 47–80%+ cost reductions from the combination of prompt caching, difficulty-based routing, batch APIs, output-token discipline, and semantic caching, provided metering and quality gates are in place first. aisuperior

Drift monitoring in generative applications. Production ML platforms document the baseline → monitor → Wasserstein/centroid/classifier comparison → LLM-as-judge classification loop for prompt and embedding drift. apxml

(These examples are drawn from public engineering material and research; specific internal numbers at Google, Meta, etc., are rarely published in full and are not invented here.)

Best Practices

1. Instrument before you optimize. Ship OpenTelemetry (or equivalent) spans and metrics for every LLM and retrieval call on day one. medium

2. Separate TTFT from TPOT in dashboards and SLOs. They fail for different reasons and need different fixes. tianpan

3. Attribute cost per feature and per model. Make the top-10 cost routes visible to the whole team. jobsbyculture

4. Prefer prompt caching + routing + output discipline before fine-tuning. They are cheaper to operate and reverse. jobsbyculture

5. Treat the semantic cache as a product surface. Version it, set TTLs, monitor hit rate and quality, and allow per-request bypass. dev

6. Run continuous, sampled evaluation. Attach judge scores to traces so quality regressions appear beside latency regressions. mlflow

7. Design for graceful degradation. Timeouts, fallbacks to smaller models or cached answers, and explicit budget guards keep the service usable when a provider or GPU pool degrades. community.ibm

8. Revisit quantization and batching under real traffic. Synthetic benchmarks miss the prompt-length distribution and concurrency patterns of production. digitalocean

9. Baseline embeddings and watch drift. Schedule the comparison; do not wait for user complaints. docs.aws.amazon

10. Keep human-readable runbooks. When p99 TTFT spikes, the on-call engineer should know whether to look at queue depth, prompt length, cache hit rate, or provider status—without reverse-engineering the dashboard.

Interview Questions

Beginner

Q1. What is TTFT and why does it matter for chat UIs?
A. Time to First Token—the delay until the first output token arrives. It determines how quickly a streaming interface feels responsive. It is dominated by queueing and the prefill phase. docs.anyscale

Q2. How is TPOT calculated?
A. $(\text{E2E latency} - \text{TTFT}) / (\text{output tokens} - 1)$. It measures average inter-token time during decode. handbook.modular

Q3. Name three metrics you would put on an LLM service dashboard.
A. p95/p99 TTFT and TPOT (or E2E), token usage and estimated cost per request/feature, cache hit rate, error rate. 21medien

Intermediate

Q4. Explain the difference between prompt caching and semantic caching.
A. Prompt caching reuses the computed state of a long stable prefix (provider-side, token-price discount). Semantic caching embeds the query and returns a previous answer when meaning is similar (application-side, avoids the call entirely). jobsbyculture

Q5. How would you detect embedding drift in a RAG system?
A. Store a baseline distribution of embeddings; periodically compare with production embeddings using Wasserstein distance, centroid shift, or a classifier two-sample test; alert and then sample + LLM-judge the change. apxml

Q6. A service has great average latency but users complain. What do you look at?
A. Percentiles (p95/p99), the gap between p50 and p99, queue depth, and whether long prompts or cold starts create a heavy tail. tianpan

Senior

Q7. How do you set an SLO for a streaming LLM API?
A. Separate SLIs for availability, p95/p99 TTFT, p95/p99 TPOT (or E2E), and a quality SLI from sampled judges or user feedback. Use multi-window burn-rate alerts on the error budget. Include fallback behavior when the budget is exhausted. learnaivisually

Q8. Design a cost-reduction plan that must not regress quality more than 2%.
A. (1) Meter cost by route; (2) enable prompt caching; (3) build a 200–500 example eval set; (4) introduce difficulty routing with the small model only when it scores ≥98% of the large model on the eval; (5) add semantic cache with a high threshold and quality monitoring; (6) tighten max output tokens; (7) gate every change on the eval and on online quality metrics. jobsbyculture

Q9. Prefill and decode have different bottlenecks. How does that affect cluster design?
A. Consider disaggregating pools, using continuous batching and paged KV cache, prefix-aware routing / KV sharing, and autoscaling policies that react to queue time (TTFT) vs decode load (TPOT) separately. medium

Exercises

Easy

1. Wrap any single LLM API call you already have with timing code that prints TTFT, approximate TPOT, token counts, and cost. Compare two different prompt lengths.
2. List every LLM and embedding call in a small app you own. Draw the span tree you would create with OpenTelemetry.

Medium

3. Implement a minimal semantic cache (in-memory or Redis + any embedding model) in front of a toy Q&A app. Sweep the similarity threshold from 0.70 to 0.95 and plot hit rate vs a hand-labeled correctness score.
4. Create a Grafana (or equivalent) dashboard with histograms for TTFT and TPOT and a counter for cost by feature. Feed it from a local OpenTelemetry collector.

Hard

5. Build a two-tier router (small model → large model) with an LLM-as-judge or cheaper classifier as the gate. Measure cost and quality on a 200-example set before and after.
6. Collect one week of production (or realistic synthetic) prompt embeddings. Compute daily centroid shift and a classifier AUC against the first-day baseline. Define an alert threshold and a short runbook.

Project

7. Production-readiness mini-project. Take an existing RAG chatbot and add: (a) full OpenTelemetry instrumentation, (b) prompt caching where supported, (c) semantic cache with metrics, (d) cost dashboard, (e) p95 TTFT/TPOT SLOs with alerts, (f) a weekly embedding-drift job. Write a one-page design doc that explains the SLOs and the cost budget.

Cheat Sheet

Formulas

Default starting thresholds

Glossary

References

The following sources were used to ground the metrics, formulas, and practices in this chapter. Prefer primary documentation and recent engineering write-ups when you implement.

• LLM latency metrics (TTFT, TPOT, ITL, E2E) and formulas — NVIDIA NIM benchmarking, independent latency decomposition guides, and inference benchmarking references. roeybc
• Production scaling, SRE practices for LLMs, quantization, batching, semantic caching, model tiering — scaling and deployment guides. linkedin
• AI/LLM observability metrics, OpenTelemetry instrumentation patterns, cost attribution — observability engineering articles. youtube
• Semantic caching design and thresholds — production caching write-ups and AWS semantic cache material. youtube
• LLM cost optimization playbooks (prompt caching, routing, batch APIs, output shaping) — 2025–2026 engineering guides. codezilla
• Embedding and data drift detection (Wasserstein, centroid, classifier tests, LLM-as-judge follow-up) — drift monitoring references. docs.aws.amazon
• vLLM production stack, KV-cache sharing, prefix-aware routing — project and conference material. youtube
• Scaling laws background — Kaplan et al., “Scaling Laws for Neural Language Models” (2020). arxiv
• MegaScale and large-scale training systems context (observability at training scale). arxiv
• Additional enterprise deployment and multi-agent observability discussions. medium

Official documentation to keep open while implementing

• OpenTelemetry GenAI semantic conventions and SDK docs
• Your model provider’s prompt-caching and pricing pages (OpenAI, Anthropic, Google, Amazon Bedrock, etc.)
• vLLM / TensorRT-LLM / chosen inference engine metrics documentation
• Prometheus / Grafana histogram and exemplar best practices
• Provider-specific batch API documentation