What can’t be tracked can’t be governed

Translated from the original Spanish article.

We built an AI agent whose only job was to direct the others and give them judgment. It worked. And yet we scrapped it.

As AI agents take on more complex work within a company’s processes, the limiting factor isn’t the technology, but governance.

AI makes “good” decisions, good meaning logical, well-grounded decisions that align with the purpose of what’s being done.

But that isn’t enough, because what’s missing is judgment.

The agent doesn’t infer the company’s culture. Culture in a broad sense, the way we humans experience it. Nor can it distill context the way we do.

And here we run into the real bottleneck: as agents become more integrated into a team’s tasks, ever more autonomous, ever more capable (there’s a lot to be said about that), keeping track of every task and every decision isn’t just inefficient, it’s unfeasible.

The answer is to formalize judgment, properly define the governance framework, spell out the guidelines and define processes in a way we never did before.

When we faced this at Kocoro, we came up with an approach we found interesting: creating a management agent whose explicit role was to bring judgment to the work of the other agents. Centralizing the instructions in a single agent that would keep alive and present the compass that guides judgment in the decisions that inevitably come up at work every day.

The other agents were to report to this “director,” who would accompany them and set the course for the options they chose.

In addition, each agent has a method built into its system for drawing lessons from its interactions and from the instructions it receives from the team, or from the “AI director.” This lives in its persistent context, so they always keep it in mind.

It made sense. But something happened that we hadn’t foreseen.

The interactions between agents are even harder to follow and control than the work each one does.

The conversations between the agents and the director (via Discord, like the rest of the team) became an unfathomable sea for any of us.

Even though the idea could work—and it did, most of the time, and in fact it objectively improved the work they were doing—it wasn’t a viable path.

Our next step in governance, on the other hand, is striking the right chords.

We understood that what can’t be tracked can’t be governed.