Week 11 Building and Evaluating AI Systems

Bounded Agents and Safe Tool Use

An agent combines a model with tools, state, observations, and a stopping rule. The important design question is not whether it can act, but how little authority it needs and how failure is contained.

Core question: When may a probabilistic model act rather than only answer?

ProposeThe model selects a bounded next action from an allowlist.

ValidateDeterministic code checks schema, authorization, limits, and confirmation.

Observe and stopThe system records tool output, updates state, and stops on success, limit, or uncertainty.

By the end

Trace an agent loop across goal, plan, tool call, observation, and stop.
Validate tool names, arguments, permissions, and outputs with schemas.
Separate read, propose, simulate, confirm, and execute authority.
Test prompt injection, repeated calls, tool failure, and stopping behavior.

Course progression

Build one evidence chain

This week uses the course-wide sequence: frame the use, establish a baseline, build or compare, evaluate failure, document boundaries, and decide.

framebaselinebuildevaluatedocumentdecidemonitor

Connection from the course

Week 12 compares whole-system options and turns accumulated evidence into governance, documentation, and release decisions.

Essential concepts

Understand the parts before combining them

The core lane focuses on transferable judgment. Optional formal or engineering depth belongs in the companion notebook's stretch lane.

Agent loop

A goal becomes a sequence of model proposals and environmental observations. Every cycle expands the opportunity for error and cost.

Tool contract

A tool needs a fixed name, typed arguments, validation, permission, timeout, idempotency, and a documented side effect.

Authority ladder

Reading, proposing, simulating, asking confirmation, and executing are distinct capabilities. Most educational agents should stop before irreversible action.

Mathematical intuition

One relationship worth keeping

\[\text{risk exposure}\approx \text{failure probability}\times\text{impact}\times\text{reachable authority}\]

Reducing reachable authority can lower exposure even when model error cannot be eliminated. Limits, confirmation, sandboxing, and read-only tools are architectural controls.

Stretch: what the notation leaves out

A more accurate model is not a security boundary; least privilege and deterministic validation limit what a mistaken or injected proposal can actually do.

Interactive explorer

Change assumptions and inspect the decision

Change tool permissions, maximum steps, and confirmation coverage. Estimate whether the agent remains bounded.

Tool permission level 3Maximum steps 6Actions requiring confirmation 8

Current signal—

Recommended response—

Risk grows with permissions and repeated steps. Confirmation helps only when it covers meaningful side effects and the user receives enough evidence to decide.

Evidence workflow

Move from a claim to a decision

Use this order in the chapter, notebook, and project record. Skipping an earlier step weakens every later claim.

1. Define one narrow goal, non-goals, allowed state, and stop condition.
2. Expose only fixed tools with typed schemas and least privilege.
3. Treat retrieved and tool-returned text as untrusted data, not instructions.
4. Log proposals, validations, calls, observations, failures, and confirmations.
5. Test injection, malformed arguments, loops, repeated calls, timeout, and cancellation.

Weekly evidence artifact

Agent trace with schema validation, injection tests, stopping behavior, and confirmation boundary

Project connection

Explore the PBL2 bounded-agent option using simulated or read-only tools, explicit confirmation, and adversarial traces.

Failure analysis

Deliberately look for the claim's boundary

A failure case is useful when the setup, expected behavior, observation, severity, and response are recorded.

Indirect prompt injection

Failure: A retrieved document tells the model to ignore the task and call a sensitive tool.

Evidence: The trace shows untrusted content changing the proposed action.

Response: Separate instructions from data, restrict tools, validate intent, and require confirmation.

Excessive agency

Failure: A calendar assistant can also send messages and delete events without need.

Evidence: The permission inventory exceeds the stated goal.

Response: Reduce tools to read/propose and gate all side effects.

Loop and cost failure

Failure: The model repeats a failing call until budget or rate limits are exhausted.

Evidence: Trace shows unchanged state across steps.

Response: Use step, retry, time, and cost limits plus a no-progress stop.

Use and non-use

Keep authority proportional to evidence

Intended use

Use bounded agents for reversible, inspectable, low-consequence workflows with fixed tools.

Do not use

Do not expose arbitrary code, shell, unrestricted files, secrets, or consequential actions without explicit confirmation.

Human responsibility

A person authorizes side effects, reviews ambiguous state, and can stop or undo the workflow.

Small Python demonstrations

Predict, run, and interpret

Each button calls one fixed, allowlisted computation. Use the notebook for longer experiments and saved evidence.

Validate a tool call

allowed = {"search": {"query"}, "draft": {"topic"}}
call = {"tool":"search", "args":{"query":"policy"}}
valid = call["tool"] in allowed and set(call["args"]) == allowed[call["tool"]]
print("valid:", valid)

output

Run this fixed example to compare your prediction with the result.

Stop a no-progress loop

observations = ["not found", "not found", "not found"]
repeated = len(observations) >= 3 and len(set(observations[-3:])) == 1
print("stop for no progress:", repeated)

output

Run this fixed example to compare your prediction with the result.

Enforce confirmation

action = {"name":"send_message", "side_effect":True, "confirmed":False}
decision = "execute" if (not action["side_effect"] or action["confirmed"]) else "request confirmation"
print(decision)

output

Run this fixed example to compare your prediction with the result.

Check your understanding

Ten questions with standard answers

Answer before opening each panel. A good answer connects the concept to evidence, failure, and a bounded decision.

1. When may a probabilistic model act rather than only answer?

Standard answer: A probabilistic model may act only through least-privilege tools, deterministic validation, explicit limits, auditable state, and human confirmation proportional to consequences.

2. What is the role of agent loop in this chapter?

Standard answer: A goal becomes a sequence of model proposals and environmental observations. Every cycle expands the opportunity for error and cost.

3. Why does tool contract require evidence rather than intuition?

Standard answer: A tool needs a fixed name, typed arguments, validation, permission, timeout, idempotency, and a documented side effect.

4. How should a practitioner use authority ladder?

Standard answer: Reading, proposing, simulating, asking confirmation, and executing are distinct capabilities. Most educational agents should stop before irreversible action.

5. What does the chapter's main formula clarify—and what does it not prove?

Standard answer: A more accurate model is not a security boundary; least privilege and deterministic validation limit what a mistaken or injected proposal can actually do.

6. What should change in the explorer as its risk or complexity controls increase?

Standard answer: Risk grows with permissions and repeated steps. Confirmation helps only when it covers meaningful side effects and the user receives enough evidence to decide.

7. How should the system respond to: Indirect prompt injection?

Standard answer: Separate instructions from data, restrict tools, validate intent, and require confirmation.

8. What evidence reveals the failure called Excessive agency?

Standard answer: The permission inventory exceeds the stated goal.

9. When should the system not be used or allowed to proceed?

Standard answer: Do not expose arbitrary code, shell, unrestricted files, secrets, or consequential actions without explicit confirmation.

10. How does this week prepare the next stage of the course?

Standard answer: Week 12 compares whole-system options and turns accumulated evidence into governance, documentation, and release decisions.

Terminology

Glossary

Agent: A system that uses a model to select actions in an environment over multiple steps.
Tool: A fixed external function the agent may call.
Schema validation: Checking that a call has an allowed name, fields, types, and values.
State: The recorded task information that changes across steps.
Memory: Selected information retained across turns or tasks.
Least privilege: Granting only the minimum authority needed for the goal.
Prompt injection: Untrusted content attempting to redirect model behavior or tool use.
Side effect: A change to external state, such as sending, deleting, booking, or purchasing.
Idempotency: A property allowing repeated execution without unintended additional effects.
Stopping rule: A deterministic condition ending the loop on success, failure, limit, or uncertainty.

Continue learning

Key references

These primary papers, standards, or official technical documents anchor the chapter. Product names and current legal timelines should be rechecked when used in a real project.