Data & Intelligence

Real customer prompts paired with gold-label answers held out of every prompt iteration, fine-tune, and few-shot example. Pass rate is the metric; benchmark scores are not. Rebuilt every quarter to track the customer's evolving task surface.

RAGAS-style axes computed on the customer's own corpus and questions, not the published benchmark dataset. Recall@k catches missed evidence; faithfulness catches hallucination; citation correctness catches cited-but-unsupported claims.

Every tool call is logged with the model's reasoning, the tool name, the arguments, and the result. Per-tool success rate, steps-to-completion vs optimal, cost-per-task. Failures categorized: wrong tool, wrong args, wrong sequencing.

Held-out adversarial test set covering known prompt injection patterns, PII probing, and jailbreak attempts. Measured as percentage of attacks blocked. Refreshed monthly from the OWASP LLM threat corpus and current jailbreak research.

Two-sample distributional shift, no assumption on shape. Sensitive to median + tail differences. Run per-feature per-day, flag p-value < 0.01 after Bonferroni correction.

PSI < 0.1 stable, 0.1-0.25 monitor, > 0.25 retrain. Industry standard for credit risk and operational scoring models. Cheap to compute, easy to chart.

Works on non-normalized distributions and ordinal features where KL-divergence misbehaves. The right primary metric for embedding-space drift in RAG and recommender pipelines.

Cumulative-sum test for shift in mean of a streaming series. Fires faster than batched tests when a concept drift starts; tuned with magnitude + threshold per channel.

Bins predictions by confidence, measures gap between confidence and accuracy per bin, weighted by bin population. ECE > 5% means the model lies about how confident it is.

Penalizes both miscalibration and inaccuracy in one number. Lower is better. Used as the headline metric when stakeholders only have time for one.

Divides pre-softmax logits by a learned scalar T. Fits in minutes on a held-out set, no retraining. First calibration we try on any classification head.

Fits a piecewise-constant monotonic function from confidence to accuracy. Heavier than temperature scaling but handles non-sigmoid distributions and pre-softmax output.

New model receives production traffic, outputs are logged and compared offline against the champion. No user sees the new model's output. The cheapest insurance policy in ML deployment.

Small percentage of real production traffic to the new model. Gates between stages are eval-line + drift signal + business-metric guardrails. Roll back within seconds, not hours.

Equal split with the success metric committed before launch. Pre-registration prevents moving goalposts post-hoc. Sample size derived from power analysis, not the time the PM wants to wait.

Traffic allocation continuously shifts toward better-performing arms. Faster convergence than A/B for high-variance metrics. Used when reward signal is dense and we can afford exploration.

Agent as a directed graph of nodes with shared state and checkpoints. Right pick when control flow has branches, retries, and human-in-the-loop interrupts. Time-travel debugging via checkpoints; the graph is the spec.

Prompts and chains are Python modules. The optimizer compiles them against a training set with a metric. Right pick when the prompt space is large and we'd rather optimize than hand-tune. The compile step is the contract.

Lightweight successor to Swarm. Each agent is a function with tools and a hand-off rule. Right pick when the topology is shallow and the routing is the interesting part. Less expressive than LangGraph but a much smaller surface.

Function calling against the provider SDK, no framework abstraction. Right pick when the agent is one model + a handful of tools + no multi-step routing. Fastest to ship, easiest to audit, and where most production agents actually end up.

Tools defined as JSON schemas, model returns a structured tool-call object. Provider-native on OpenAI, Anthropic, Bedrock, Vertex. Schema validation is the first defense against malformed args; we run it pre-execution every time.

Open protocol for connecting agents to external systems. Vendor-neutral, so the same tool surface works across Anthropic, OpenAI, and self-hosted models. Right pick when the tool catalog is shared across multiple agents.

Multiple tool calls executed in parallel from one model turn. Cuts agent latency on read-heavy workloads (lookups, retrievals, joins). Requires the framework or runtime to deduplicate, aggregate, and re-inject into context cleanly.

Agent writes code, runtime executes it in a sandboxed environment with bounded CPU, memory, and network. Right pick when the action space is too large for a static tool catalog (data analysis, transformation, ad-hoc scripts).

Whatever fits in the model's context window for the current turn. The cheapest memory and the only one with zero retrieval latency, but it's lost the moment the turn ends. Budget it like a register file.

Model writes intermediate steps to a structured scratchpad inside the same context window. Improves multi-step reasoning quality, makes the chain auditable, and lets downstream tools key off named intermediate values.

Long conversations compressed into a running summary that lives outside the immediate context window. Trade-off: the summary loses fidelity over time. We re-derive from raw transcript when the summary stops being trusted.

Episodic and semantic memory indexed by embedding. Retrieved by relevance to the current state, not chronology. Persists across sessions. The right home for the kind of memory that has to outlive any single conversation.

One supervisor classifies the request and routes to one of N specialist agents. Specialists return to the supervisor, supervisor returns to user. Easiest topology to reason about and the right pick for most production multi-agent systems.

Output of agent N is input of agent N+1, in a fixed order. Right pick when the workflow is genuinely linear (e.g., extract -> classify -> summarize -> publish). Less flexible than supervisor but trivially testable and resumable.

Each agent decides which peer to hand off to. No supervisor; routing is emergent. Right pick when the right specialist isn't knowable from the request and has to be discovered. Harder to debug, easier to extend.

N agents independently attempt the same task, a judge model or majority vote selects the answer. Right pick when one wrong answer is expensive and the marginal compute cost of N attempts is acceptable.