How to monitor AI agents in production: cost, failures, accountability

Tracing tells you what the model said. Monitoring tells you what the agent cost, whether it finished, and whether anyone noticed.

The tooling for AI agents has grown up around one question: what did the model do? Traces, prompts, token counts, evals. All useful for the person building the agent. Almost none of it answers the questions the person running the company asks: what did it cost this week, did it finish, and did it break anything?

Those are operational questions, and they want an operational answer. An agent in production is a service like any other. It should report each run the way a deploy reports itself: it started, it took this long, it cost this much, it succeeded or it did not. Once it does, it can be watched with the same rules as everything else.

Cost first, because it is the one that surprises people. Model spend is a curve, not a number, and a curve that doubles on a Tuesday because a prompt changed will not appear in any dashboard until the invoice. The signal to watch is spend per run and spend per day against your own last thirty days, with the provider's balance alongside so a depleting credit is news before it is an outage.

Failures second. An agent that fails silently is worse than one that crashes loudly, because the downstream work simply does not happen. A failed run should be a critical event, raised into the same incident list as a failed payment, with the run's own summary as evidence.

Then accountability, which is the part nobody plans for. When an agent can act on your systems, it should also be able to report on itself, and it should not be able to hide. The right design is asymmetric: an agent may add signal, never remove it. It can report its run, it can propose a rule, it cannot resolve or mute an incident a human has not seen.

This is where the Model Context Protocol becomes useful for operators rather than just for developers. Give your coding agent read access to the operational picture and it can answer 'did my deploy break anything' from inside the editor. Give it a scoped write token and it can report its own runs. Give it nothing destructive, ever.

Setup is smaller than it sounds. A webhook or an MCP token for the agent to report into. A few signals: run count, failure count, cost, duration. Two rules: failed runs above a threshold, spend above a threshold. A morning brief that mentions the agents by name.

InternalStatus ships all of that, including the MCP server with the asymmetry built in. Connect an agent, report a run, and watch it appear next to the payments and deploys it was supposed to help with.