> For the complete documentation index, see [llms.txt](https://blackprince001.gitbook.io/flowbench-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://blackprince001.gitbook.io/flowbench-documentation/project/context.md). # Glossary FlowBench is an internal, local-first toolkit for testing API endpoints and multi-step flows: one canonical flow definition, executed under four profiles by one Go engine. This glossary is the project's ubiquitous language; use these terms exactly, in code, docs, and issues. ## Language ### Authoring **Endpoint**: A declared, reusable target — protocol, method/operation, URL or address template, default headers, auth requirement. *Avoid*: route, API **Flow**: An ordered, optionally branching sequence of Steps; the unit of authorship. Written once in YAML or Python, tested four ways. *Avoid*: test case, collection, journey **Step**: The atomic unit of a flow. Types: `call` (HTTP), `graphql`, `ws`, `grpc` (unary), `wait`/`poll-until`. Can extract values, assert conditions, and carry a retry/backoff policy. *Avoid*: request, task **Prompt observation**: An LLM call made by the flow's own SDK code inside a `logic` step, wrapped by the Python SDK's prompt-observation API (`ctx.prompt(...)` … `p.record(...)`). FlowBench never makes the call or sets model behavior; it captures the prompt and completion (always), hashes the prompt's identity, emits a `prompt` span, and can pace and time-bound repeated calls. *Avoid*: prompt step, eval, LLM test **Prompt variant**: A label on an observation naming which prompt version the author's code used, giving it its own structural span identity (`name@variant`) so folding, metrics, and diffs stay per-variant. What varies is decided entirely by the author's code. *Avoid*: A/B test **Pace guard**: An optional client-side rate ceiling on an observation (e.g. `20/m`, optional burst allowance), coordinated by the engine across VUs and workers, so calls repeated N times don't trip a provider's rate limit. Complements the profile-level arrival cap; throttles that still occur classify as `throttled`. *Avoid*: rate limiter (that's the target's) **Completion**: The LLM output recorded by a prompt observation — captured, asserted on in plain Python, and diffed. *Avoid*: output, generation **Extraction**: Capturing a value from a step's response (JSONPath; later XPath) into a flow variable for injection into later steps. **Assertion**: A per-step condition on status, latency, headers, or body; failure behavior is configurable (`abort_flow | abort_run | record`) with mode-aware defaults. **IR (canonical flow representation)**: The single structure both authoring surfaces compile to and the only thing the executor accepts. *Avoid*: AST, intermediate format ### Configuration **Profile**: The execution contract that turns one flow into any of the four test categories — mode (`integration | system | load | stress | soak`), VUs, ramp shape, duration or iteration count, thresholds, optional arrival cap. *Avoid*: config, test type **Scenario**: One or more flows bound to a profile, a target config, and data pools. The runnable unit. *Avoid*: suite **Target Config**: A lightweight named file (local/dev/staging) carrying base URLs, VU/RPS ceilings, and optionally an agent address. Never carries credentials. *Avoid*: environment **Data Pool**: A fixture source (CSV/JSON) with a distribution policy (`unique-per-vu`, `round-robin`, `random`) so concurrent VUs draw distinct rows; also the sanctioned seeding mechanism. *Avoid*: dataset, seed file **Arrival cap**: An optional, self-imposed request-rate ceiling on a profile, enforced by the planner ahead of the target, for testing at a known rate limit rather than only discovering one by flooding. ### Execution **VU (virtual user)**: One concurrent executor of iterations — a goroutine — with isolated variables, cookie jar, and data rows. **Run**: One execution of a scenario, producing aggregate metrics, threshold evaluations, per-iteration traces, folded flame data, and (if an agent is attached) target resource series. Records initiator, target, and flow-file git commit. **Iteration**: One VU's single pass through a flow, recorded as one trace. **Outcome**: The classification of a response at the point of assertion: `ok | failed | throttled | skipped`. **Throttled**: The outcome class for rate-limit responses (HTTP `429`, gRPC `RESOURCE_EXHAUSTED`, or an author-mapped status), tracked separately from `failed` via its own `throttle_rate` metric. *Avoid*: rate-limit error **Knee point**: The concurrency level at which thresholds begin to fail during a stress ramp, classified as **degraded** (real capacity limit) or **throttled** (rate limiter engaging). **Coordinated omission**: The measurement error where a slow target quietly suppresses request attempts, flattering latency stats; the engine must account for it. ### Observation **Span**: The atomic unit of tracing — a named, timed node (step, protocol phase, extraction, assertion, poll or retry attempt) with a parent, start offset, duration, self-time, and outcome. Span names carry structural identity so folding across iterations and runs works. **Trace**: One iteration's spans assembled into a tree in causal order; the input to the waterfall view. **Flame graph (of a flow)**: A fold of many traces — spans with the same structural name collapsed and summed — rendered as width-proportional time, per iteration, per run, or cumulatively across runs. Answers "where does aggregate time go." **Waterfall / trace view**: A causal, per-iteration rendering of one trace's spans in start-offset order, like a browser performance panel. Answers "what exactly happened, in order, in this one iteration." **Prompt diff**: The results-server comparison of captured completions — variant vs variant within a run, or same observation/variant against a baseline run — rendered as a text diff (structural for JSON), with the prompt's own diff shown when the prompt hash changed between runs. The v1 answer to "how are my prompts doing"; scoring is deliberately not one. *Avoid*: eval score **Run store**: The on-disk directory of run artifacts (folded flame data, raw trace trees, agent series, aggregates) that the results server and CLI read. No retention machinery; the user owns it. **Results server**: The small embedded server (`serve`, in the spirit of `go tool pprof -http`) reading the run store locally. Not a web app: no accounts, no persistence of its own, no write path beyond abort. **Agent**: The small standalone binary on target hosts streaming resource metrics (CPU, memory, network, descriptors) into a run, keyed to the run ID. Fails open — a dead agent never affects the run, only the overlay. **Collector**: The engine component that streams spans into the two storage tiers, ingests agent and engine self-metrics series, applies redaction, evaluates thresholds/trends, and classifies knee points. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://blackprince001.gitbook.io/flowbench-documentation/project/context.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.