Building an Executive Admin Portal for AI Agents: The Control Plane Behind Connect

Most multi-tenant products have two levels of admin: workspace admins who manage their own team, and a super-admin who can fix anything. Once AI agents are in the product, there's a third job that neither covers. Someone has to see every agent across every customer, read any trace, change a prompt, approve a knowledge document, pause an agent for one client and decide which tenants get which features. That person shouldn't need database access or an SSH session to do it.

In Connect, that's the executive admin portal. We've written separately about what governing AI agents involves as a checklist. This article is about building the control plane: how it's scoped, how it's gated, what's in it, and why we gave executives a read-only MCP server so they can ask Claude about agent operations instead of clicking through dashboards.

Org-wide on purpose

Everything else in Connect is workspace-scoped. A member route has the workspace id in its path, and one authorization chokepoint checks membership, feature entitlement and role level before a handler runs. The admin portal deliberately breaks that pattern.

Its routes have no workspace in the path and no membership hook. The handlers query the agent ledger, configs and evals across all tenants, because the questions are org-wide: "what did the Info Sheet cost us last month across every client?" or "show me every failed Schedule Builder turn this week."

That's a significant design choice, and it puts all the weight on the gate. In a workspace-scoped route, a missing check usually exposes one tenant. Here it would expose all of them. So we made the gate as narrow and boring as we could.

The gate: super-admin AND an operator-domain allowlist

There's one permission for the whole surface, agents.admin, and it's checked in one function. A caller passes only if both are true:

1. They're a platform super-admin. That flag is local to Connect and granted out of band, never synced from SSO.
2. Their email, which comes from the trusted SSO identity provider, is on the operator's domain allowlist.

A super-admin on any other domain is refused. So is an operator employee who isn't a super-admin. The gate has its own tests for exactly those three cases.

Why both? Super-admin alone is too broad. It's granted for support and operations work that shouldn't include reading every customer's agent conversations. The domain check alone is too broad in the other direction. Together they describe a small, specific group: the operator's own platform owners.

Every route in Connect is declared in a single manifest with its authorization level, and a test asserts that the registered routes match the manifest exactly. The executive level is one of the labels, so a new admin route can't be added without a reviewer seeing it labeled as executive-only. There are around eighty executive-only routes in the manifest today. The React shell hides the portal for non-executives too, but that's only for convenience. Every /api/admin route is checked on the server independently.

A few platform operations, such as approving App Builder SQL queries and managing workspaces, live under the separate super-admin tier rather than the agent-governance gate.

What's in the portal

The navigation groups pages by what an executive is trying to do, using plain language rather than internal module names. The routes stay stable so deep links keep working.

A few of these are worth a closer look.

Overview and the attention list

The overview reads a single payload and builds a "needs attention" list from it with no extra queries. We restricted it to signals that are clear 0-to-1 rates (error rate, groundedness and cost spikes), so it can't raise a false alarm because an eval score was on an unfamiliar scale. Each agent gets green, amber or red using the same thresholds.

Per-agent configuration with versions

Each agent's detail page has four tabs, in the order we think decisions should be made: configuration (model, child model, thinking level, system prompt), knowledge and skills, tools, then activity.

Configuration is versioned. An edit creates a draft that changes nothing for users. Publishing validates any model override against the live provider first, so a bad deployment name fails at publish time and never in front of a user. An archived version can be republished as a rollback. Each ledger turn records the version that governed it, so a quality change can be traced back to a config change.

Knowledge documents follow the same discipline. An edit to a published document becomes a pending draft until someone approves it. Each document can be switched between full injection into the prompt and per-turn retrieval, and toggled on or off without deleting it. An agent can also be bound to specific data sources. An empty binding means it grounds against the whole workspace.

Logs and traces

The logs page lists conversations and turns with filters for agent, workspace, project, user, model, status and user feedback. Opening a turn shows its spans, model calls, tool calls and arguments, reasoning summaries, and links to the exact file versions it cited. Those links resolve through the server, which first checks that the trace actually references that file version, and then redirects to a short-lived download URL with caching turned off. Conversations can be exported as JSON or flat CSV for offline analysis.

Directory and per-client controls

The directory lists every client workspace with its members, spend and volume. From there an executive can pause one agent for one client, including the Updates agent, whose output goes straight to the client. The global user search stays available because someone who has signed in through SSO but hasn't been added to a workspace yet still needs to be findable.

Feature Access and Roles

Feature Access is the outermost of three RBAC gates: which features a tenant has at all. Only the app tier is switchable per tenant. Core features are always on. Features the server grants together, such as the Info Sheet and Means & Methods as one package, appear as a single control, because separate switches would just be normalized back by the server.

The Roles page edits a tenant's role builder from outside the tenant. It uses the same handlers and rules as the in-workspace routes. Only the path prefix and the gate are different. There's also an escape hatch: a workspace whose only admin has left can't be fixed from inside, because every route that grants admin requires an admin already. An executive can set the admin flag on a member directly.

MCP settings, global and per-workspace

Connect exposes an MCP server to members. Executives can change its runtime configuration without a redeploy: switch the advertised tool profile, disable individual tools, or turn off external access to the Schedule Builder agent. Per-workspace overrides can only restrict the global surface further, and the editor shows what each workspace would inherit and what's effective, so nobody has to work it out by hand. A write triggers an immediate cache refresh, so a change takes effect on the next call instead of after the cache's normal interval.

RAG browser and query approvals

The Workspace Knowledge Base page shows every workspace's indexed documents and their status, with bulk reindexing. It also shows the extracted text that actually gets chunked and embedded, which is usually what you need when an agent "can't find" something that's in the file. Documents can be excluded so agents can't retrieve them.

Query Approvals controls the App Builder's data access. When the App Builder agent writes a SQL query, it's saved as pending and can't run or be installed until a super-admin approves it. Rejecting deletes it, and the agent can propose again. Approved queries stay listed so it's always clear what apps can read.

Connector request triage

Any signed-in user can suggest a connector and upvote others. Executives move requests through a fixed set of statuses, from requested to considering, planned, in progress, live soon and live, or declined. That status change is an executive-only route, and the control appears inline on the suggested-connectors board for executives only. It's a small feature, but it turns "can you connect to X?" from an email thread into a public, ranked backlog.

A read-only admin MCP so executives can ask Claude

Dashboards answer the questions you thought of when you built them. Executives and engineers kept asking ones we hadn't thought of, like "which tools fail most for this client since Tuesday's publish?" or "compare the Info Sheet's cost per turn this month to last." So we put an MCP server in front of the same admin module.

It's a separate MCP surface from the member one, with its own OAuth issuer and metadata documents, public clients only, and S256 PKCE required. About two dozen tools, all thin wrappers over existing admin read methods: overview, traces and conversations, usage summaries, breakdowns and time series, tool usage, cohorts, the agent graph, MCP usage, provisioning, active configs and versions, insights, the autopilot audit trail, golden sets and rubrics. Every tool is annotated read-only, non-destructive and idempotent. None of them can publish a config or change a setting.

The gate is checked twice. The same agents.admin function runs when the authorization code is issued, so a non-executive never gets a code and no token can be minted. It runs again on every transport call, so a super-admin whose access is revoked loses the MCP immediately, not when their token expires. Reusing the same authorization function means the domain boundary is defined in one place.

The result is that an executive can attach the admin MCP to Claude and ask about agent operations in plain language. Claude reads the ledger the dashboards read and answers with the same data. Write actions stay in the portal, behind drafts, regression runs and the review queue.

What it cost

• A second authorization model. Org-wide routes don't fit the workspace chokepoint, so the executive gate is a separate path we have to keep narrow. The manifest test and the dedicated gate tests are how we keep it honest.
• A big surface to maintain. Dozens of pages and around eighty routes is a lot of UI for a small group of users. We accept it because the alternative, engineers running SQL against production for every question, is worse and less auditable.
• Operator-specific policy in code. The domain allowlist is a constant, not a setting. That's intentional: changing who can read every tenant's agent traces should take a reviewed code change.

Why this matters if you're building something similar

• Treat agent governance as its own permission. Don't stretch super-admin to cover it. Reading every customer's AI conversations is a different level of trust from resetting a password.
• Put every route and its gate in one reviewable list, and test that the list matches what's registered.
• Make config changes drafts by default. Validate at publish time, keep old versions for rollback, and stamp the version on every turn.
• Give per-tenant overrides restrict-only semantics. Overrides that can only take capability away are much easier to reason about than overrides that can add it.
• Expose your admin data over a read-only MCP. It costs little once the read methods exist, and it answers the long tail of questions no dashboard will cover. Check the gate at token issue and on every call.

Where to go next

• The series overview: how we built Connect
• The data behind the Logs and Usage pages: tracing LLM agents in a Postgres ledger
• What the Quality section runs: LLM evals, golden sets and an autopilot gate
• The workspace-scoped side of authorization: multi-tenant RBAC with a single authorization chokepoint
• The member MCP the admin one sits beside: building an enterprise MCP server with OAuth

Building agents your executives need to oversee? Tell us what you're working on.