Skip to slide
182 / 221

CH 15 · Observability and Evaluation · 07/08

Read as one page

Online signals

Beyond offline evals, watch production:

  • Explicit feedback: thumbs up/down, corrections, regenerations. Aggregate by feature and model.
  • Implicit signals: did the user accept the agent's proposed edit or reject it? Did they re-ask the same thing (a sign the first answer missed)? Did they abandon mid-turn?
  • Failure and degradation rates: from your traces, which tools and dependencies fail most, and is it trending.

These tell you what your eval set can't: how the agent performs on the messy distribution of real use. Feed surprising production cases back into the eval set so it keeps reflecting reality.

Finished this idea?

Your learning progress stays in this browser.