Start with the decision, not the model
The easiest way to build the wrong AI system is to begin with a model capability and search for somewhere to put it. In engineering operations, the more useful starting point is a recurring decision: one that matters, arrives with imperfect context, and currently consumes too much expert attention.
Incident response is full of these moments. Is this alert a symptom or a cause? Which recent change is worth investigating first? When should the team escalate? The opportunity is not to replace the engineer making the call. It is to reduce the distance between scattered evidence and a well-informed decision.
The real problem is signal entropy
Most teams do not lack operational data. They have logs, traces, deploy events, tickets, runbooks, chat threads, and dashboards. What they lack is coherence at the exact moment it matters. Every source speaks a slightly different language and carries a different level of trust.
A useful AI layer should compress this entropy. It should connect an anomaly to the service topology, place it beside recent changes, surface relevant precedent, and be explicit about what it does not know. That is a systems-design problem before it is a prompting problem.
- Normalize evidence before asking for interpretation.
- Preserve provenance so every claim can be inspected.
- Treat uncertainty as product data, not hidden model behavior.
Build a ladder of trust
I like to think about adoption as a ladder. First, the system observes and organizes. Then it summarizes. Next, it recommends. Only after the recommendation layer is consistently useful should selective automation enter the conversation.
Each rung has a different evaluation burden. A summary can be judged for coverage and factual grounding. A recommendation must also be judged for relevance, calibration, and operational risk. An automated action needs clear permissions, reversibility, and an audit trail. Skipping these distinctions creates a demo that feels magical and a production system that feels unsafe.
Evaluation is part of the architecture
Before building the polished interface, assemble a small but representative set of past situations. Include the ordinary cases, the noisy cases, and the incidents where experts initially disagreed. Ask what a strong answer would have needed to contain and which mistakes would have been costly.
The goal is not a single accuracy number. It is a map of failure modes. Does the system miss an important deploy? Does it overstate a weak correlation? Does it recommend the same playbook regardless of context? This map should shape retrieval, permissions, interface design, and rollout strategy.
The direction
The most promising engineering AI is quiet. It does not perform intelligence for the user. It makes the surrounding system easier to understand, gives experts a faster path through ambiguity, and leaves a clear trail of evidence behind every suggestion.
That is the standard I am working toward: less time reconstructing context, more time applying judgment, and automation that expands only as trust is earned.