Vocabulary for building AI agents that plan, act, and recover from failure in production systems.
A backup model an application switches to automatically when the primary model is unavailable, rate-limited, or returns a bad response.
The ability to see what an agent actually did step by step after the fact, through logs, traces, and metrics, rather than just its final answer.
Storing a prefix of a prompt (like a long system prompt or document) on the provider's side so repeated calls skip re-processing it, cutting cost and latency.
A cap on how many requests or tokens an agent can send in a given window, enforced by the API provider or the app itself, to avoid overload or runaway cost.
Sending a model's response back piece by piece as it's generated, instead of waiting for the whole thing to finish before showing anything.