Structured outputs, tool calling & guardrails
LLM applications fail when free-form text is treated like a trusted API response. Use structured outputs when the model must return data, tool calling when it needs external data/actions, and guardrails when runtime policy must block or redirect unsafe behavior.
Structured output lifecycle
Key distinction: schema validation checks shape. Semantic validation checks truth and safety.
JSON mode vs structured output vs tool calling
| Mechanism | Best for | Limitation |
|---|---|---|
| JSON mode | "Return parseable JSON." | Does not guarantee your schema. |
| Structured output | Extracting or generating typed data. | Valid values can still be factually wrong. |
| Tool calling | Fetching external data or taking actions. | Tool must be validated, authorized, rate-limited, and audited. |
| Guardrail | Blocking risky inputs, outputs, or tool calls. | Must be part of layered controls, not the only defense. |
Tool calling architecture
The safest tools are small, typed, boring, and auditable.
Tool design examples
| Unsafe tool | Safer tool |
|---|---|
run_sql(query) | get_customer_orders(customer_id, limit) |
send_email(to, subject, body) with no approval | draft_email(ticket_id, recipient_id) then human approval to send |
delete_file(path) | request_file_deletion(file_id, reason) with policy review |
browse_web(url) unrestricted | fetch_allowed_domain_article(article_id) |
shell(cmd) | Purpose-built build/test/deploy actions with allowlists |
Guardrail positions
| Guardrail | Catches |
|---|---|
| Input | Abuse, prompt injection, unsupported intent, PII oversharing. |
| Output | Sensitive leakage, unsupported claims, policy violations, malformed response. |
| Tool input | Overbroad args, unauthorized ids, dangerous side effects. |
| Tool output | Secret leakage, injected instructions, unexpected payload shape. |
Repair loops
Use repair loops carefully:
- Cap repairs, usually one attempt.
- Log the original invalid output and repair reason.
- Do not repair policy violations into allowed actions.
- Prefer deterministic validation errors over vague "try again."
- Fall back to clarification, refusal, or human review after exhaustion.
Interview questions
1. Why is structured output not enough for safety?
- It controls shape, not intent, authorization, or factual correctness.
2. What happens after the model emits a tool call?
- The app validates schema, checks policy and permissions, executes the tool if allowed, appends the observation, and asks the model to continue or answer.
3. Where do you put human approval?
- Before irreversible, external, financial, legal, security-sensitive, or destructive tool execution.
4. How do you prevent tool output injection?
- Treat tool output as data, strip/label instruction-like text, validate payload shape, and keep system policy outside retrieved/tool content.
5. What is a good contract test?
- Given an input, assert the expected tool name, argument shape, permission behavior, refusal path, and final response class.
Related
LLM contracts, context & tools · Agentic production · Safety & injection
Spotted something unclear or wrong on this page?