Khali Kalpa-Young

Extracting Tacit Knowledge for Production Agents

An eval is a definition of "good" written down well enough for a machine to grade against. In a real business, "good" isn't written down anywhere. It lives in someone's reaction, in what past work actually did, and in how a person talks. So the eval comes second. First you get "good" out of wherever it lives, and each place needs its own method.

The first place is the person's reaction. Two domain experts corrected a coaching agent in a Telegram group, and each correction became a commit to the agent's own instruction file, routed by whether the agent could apply it alone, had to ask me first, or should stay a note. Then a panel of critics I built to grade runs lied to me, rating a shallow run "demo-ready and close to trial-ready." The fix was not a better judge prompt. It was turning that failure into a permanent human-labelled case and refusing to trust any critic that couldn't reproduce my verdict.

The second place is the system's residue. For a second agent, the owner couldn't describe his own process, so I reconstructed it from past campaigns and tags on old lists and proposed it back for correction. The third place is his conversation history. I mined it for every correction and acceptance and built a simulated version of him, so new versions could run against him without him present.

A business can only hand an agent the work it can check, so the eval is the edge of what you can delegate. I'll cover the infrastructure: an agent that commits to its own instructions under authority rules, hot reload into live sessions, simulated users built from transcripts, a gate-first rubric, and critics that must pass a human-labelled test before they're trusted.