How Do You Know the Agent Did Well?
The smallest eval that works: make the agent report what it did, which checks it ran, and what it did not do. A one-line decision log caught a bug in my agent.
Every time North makes a real judgment, it adds one line to a decision log: the date, the domain, and what it decided.
One day, that log caught a bug. One of my scripts reads the energy table in my journal, but it read the empty example row in the table as real data. North did not accept the value as it was. It wrote “this script has a problem” in the log and reported it to me.
Without that line, I only get a strange reminder, and I guess at the cause.
Three kinds of eval
The talk by 「大大帶我飛」 splits eval into three kinds:
The smallest eval that works
You do not need a scoring system yet. The talk gives a smallest starting point: make the agent report four things.
- What it finished.
- Which checks it ran.
- Which tools it called.
- Which risks remain, or what it did not do.
North’s decision log is a short version of that report. It is also the raw material for the meta loop in Three Loops in an Agent System. When the log grows, you can find the judgments that repeat, as Graduate NFR into FR shows.
Without a trace, you can only look at the result and guess the cause.
References:
- A talk by 「大大帶我飛」
Related: see the meta loop this log feeds in Three Loops in an Agent System, how repeated judgments graduate in Graduate NFR into FR, or go back to the series overview: Design an Agent.