Arnaldo Sepulveda
Applied AI · Intervention design

A working AI system can still be the wrong intervention

Why enterprise AI work should begin with workflow diagnosis, evidence, and intervention selection before deciding what to build.

· 12 minute read

An operational problem appears.

Someone proposes AI.

A pilot gets funded. The demo works. The system reaches production.

Months later, nobody can clearly show that the operation improved.

The model may be responding. The integration may be stable. Users may even like the interface. The engineering can be competent and the implementation can work as designed.

So where did the reasoning fail?

Sometimes it failed before implementation. The organization moved from a visible symptom to a preferred technology without establishing what caused the problem, where the constraint occurred, or what kind of intervention the evidence justified.

My previous note, AI availability is not a business case, asked whether AI should be used at all. The next question is more practical: given a real operational problem, how should an organization decide what to do about it?

The most important design decision may happen before architecture. It is the decision about what kind of intervention to make.

The symptom is not the diagnosis

Organizations experience problems as symptoms:

These observations matter. None of them identifies a cause.

High handling time could come from missing knowledge, poor search, fragmented systems, bad routing, duplicate data entry, an approval queue, incomplete telemetry, ambiguous process ownership, a training gap, or a problem that genuinely requires expert judgment.

Several mechanisms can produce the same visible outcome. One mechanism can also produce several symptoms in different parts of the operation.

More than twelve years in Technical Support and Escalations at Genesys taught me not to accept the first visible symptom as the cause. Across complex contact-center systems, integrations, migrations, go-lives, and production incidents, the reported failure was often only the first clue.

A timeout could originate in the application, the network, a data dependency, capacity, or configuration. The practical task was to narrow the failure domain with evidence.

That discipline carries into intervention selection: identify the mechanism before choosing the remedy.

Before drawing an AI architecture, draw the work

Operational problems become easier to reason about when the current workflow is made explicit.

Who starts the work? What information do they have at that point? Which systems do they touch? Where is information copied manually? Where does work wait? Where do people make decisions? Which decisions follow known rules, and which require judgment?

Then look at the transitions.

Where are the handoffs? What context is lost between them? Where do errors first occur, and where do they finally become visible? What happens after an escalation? Does the next person receive evidence, or do they have to reconstruct the investigation?

A workflow diagram is useful because it forces the conversation away from an imagined AI feature and toward the actual path of work. It can reveal that the longest delay is a queue, not an analysis task. It can show that employees spend two minutes interpreting a request and twenty minutes waiting for access to another system. It can show that a customer repeats information because identity and interaction context do not cross a channel boundary.

Those findings change the intervention space.

If the constraint is a policy approval, faster text generation will not remove it. If the constraint is missing account context, a system integration may matter more than a model. If work is routed to the wrong team, classification might help, but so might correcting deterministic routing rules or the data they depend on.

The workflow map is not the diagnosis by itself. It is a way to locate observations, decisions, queues, controls, and dependencies so competing explanations can be tested.

Establish a baseline before proposing improvement

"People say this is painful" is a reason to investigate. It is not enough to evaluate an intervention.

A baseline establishes what is happening before the system changes. Depending on the workflow, that might include handling time, resolution time, queue time, escalation rate, repeat contact, transfer count, search attempts, backlog age, rework, abandonment, analyst effort, or customer effort.

Not every operation needs every metric. The measurement should follow the mechanism being investigated.

If the hypothesis is that poor knowledge retrieval drives escalation, measure how often analysts search, what they find, where searches fail, and whether failed searches correlate with escalation. If the hypothesis is that fragmented systems extend handling time, measure the systems touched, context switching, manual copying, and time spent waiting for data.

The baseline serves two purposes.

First, it shows whether the problem is material. A frequently discussed frustration may affect very little work. A less visible delay may account for most of the backlog.

Second, it makes later evaluation possible. Without a credible before state, a successful deployment can be confused with a successful intervention.

In current Keystone research, I am applying this discipline more explicitly by examining baselines, competing explanations, retrieval behavior, authorization, and evaluation before deciding whether an intervention is warranted. The point is to preserve the possibility that the evidence leads somewhere else.

Separate the cause from the proposed mechanism

Consider a familiar statement:

Support analysts spend too long resolving known issues.

That description can support several different diagnoses.

Diagnosis A: the knowledge does not exist

The investigation was completed, but the resolution was never captured in an authoritative form. The intervention is knowledge creation, ownership, and maintenance. A model cannot reliably retrieve information that the organization never preserved.

Diagnosis B: the knowledge exists, but search is poor

The answer is present and accessible, but analysts cannot find it using the vocabulary available to them. Better indexing, metadata, lexical retrieval, semantic retrieval, or ranking may address the failure. Generation may add no value.

Diagnosis C: the knowledge is spread across systems

The resolution depends on account data, product telemetry, prior interactions, and a technical note stored in separate places. The core intervention may be integration and context aggregation. Summarization could help after the evidence is assembled, but it does not remove the fragmentation.

Diagnosis D: analysts repeatedly execute known rules

The work consists of collecting defined fields, checking fixed conditions, and selecting a known path. Deterministic automation or conventional workflow software may be cheaper, more observable, and easier to control than a probabilistic system.

Diagnosis E: the input is unstructured and variable

Customers describe the same issue in many ways, documents use inconsistent formats, or relevant evidence must be extracted from free text. AI-assisted classification, extraction, or interpretation may now be justified because the ambiguity is part of the failure mechanism.

Diagnosis F: the next action depends on intermediate evidence

The system must gather evidence, inspect what came back, choose among approved next steps, and stop or escalate when defined conditions are met. Bounded agentic behavior may be justified if variable sequencing creates enough value to offset the added state, permissions, and failure modes.

The initial symptom is the same in every case. The appropriate intervention is not.

Compare interventions before selecting one

The intervention space is wider than a feature comparison between models.

It can include process change, knowledge improvement, data correction, system integration, deterministic automation, conventional software, analytics and decision support, retrieval, AI-assisted interpretation or generation, and bounded agentic behavior.

These are not substitutes ranked from least advanced to most advanced. They address different failure mechanisms.

For each plausible option, I want to know:

A good intervention decision explains not only why one option was chosen, but why plausible alternatives were rejected.

That record does not need to become a large governance exercise. A concise decision can still state the diagnosis, assumptions, alternatives, expected effect, risks, and test. The discipline matters because implementation creates momentum. Once a team has built a pilot, evidence that points to a simpler solution can start to feel like a threat to progress rather than useful information.

Intervention selection identifies the mechanism that should address the diagnosed problem. Implementation determines how that mechanism enters the real workflow, including its interfaces, controls, authorization, observability, failure handling, and human responsibilities.

AI should earn its place in the architecture

Selecting AI for one part of an intervention does not make every part of the architecture probabilistic.

Permissions, eligibility, fixed thresholds, known routing rules, state transitions, and deterministic controls should usually remain explicit. An AI component may extract an attribute from text, but software can enforce what that attribute permits. A model may summarize evidence, but authorization should determine which evidence it can receive.

Put probabilistic behavior where ambiguity exists. Keep deterministic controls where the rule is already known.

This boundary makes the system easier to evaluate. It also makes failures easier to attribute. If one component retrieves evidence, another enforces authorization, and another generates a summary, the team can inspect which mechanism produced the observed result.

The intervention can be mixed

"AI or no AI" is often the wrong choice because a strong enterprise intervention can combine several mechanisms.

Imagine a customer issue arriving through a digital channel.

Integration gathers the account, product, and interaction context. Deterministic logic applies known rules, including which records are authorized. Retrieval handles evidence discovery. An LLM interprets semantically variable evidence and identifies unresolved questions. A human makes the consequential decision. Workflow automation records and routes the result. Observability captures what happened at each boundary.

The value does not belong to the LLM alone.

The architecture is mixed because the failure mechanisms are different. Fragmented context, evidence discovery, semantic ambiguity, known controls, consequential judgment, execution, and accountability do not call for the same mechanism.

Calling the whole intervention an AI assistant hides the architecture that makes it useful.

The right intervention is often an architecture of mechanisms, each placed where its behavior fits the problem.

Define success at four levels

Evaluation becomes clearer when success is separated into four levels.

Component performance

Did retrieval return relevant evidence? Did the classifier meet its defined accuracy threshold? Was generation latency acceptable?

System behavior

Did authorization work? Did the workflow complete correctly? Were failures handled? Was evidence sufficient for the output? Did the system stop when it should?

Workflow outcome

Did resolution time, escalation rate, analyst effort, repeat work, or another relevant operational measure improve?

Business outcome

Did the operational improvement create capacity, improve service level, reduce avoidable cost, change customer outcomes, or affect revenue or risk where the evidence supports that connection?

Component performance is not system performance. System performance is not workflow improvement. Workflow improvement is not automatically business value.

If an assistant saves time, the organization still needs a realization mechanism. Perhaps the team absorbs more demand without hiring. Perhaps backlog falls, service levels improve, or experts spend more time on novel problems. Perhaps fewer errors avoid rework.

Those are testable paths from operational change to business value. Without such a path, a time-saving estimate remains a technical or workflow claim, not a financial result.

Decide what would make the work stop

Intervention selection should include exit criteria, not only success criteria.

Before implementation, define the evidence that would justify stopping, modifying, rolling back, declining to scale, or replacing the intervention with a simpler mechanism:

I see this as part of honest evaluation. A pilot should not exist only to accumulate reasons for deployment. It should reduce uncertainty about the diagnosis, the mechanism, and the expected outcome.

Sometimes the result will support broader implementation. Sometimes it will reveal that the workflow must change first. Sometimes it will show that retrieval is enough, that a data correction removes the problem, or that the effect is too small to justify the operating burden.

Stopping under those conditions is not a failed experiment. It is the intervention process working.

The real capability is disciplined intervention selection

AI engineering is necessary when AI is the justified mechanism.

It cannot, by itself, establish that the problem was diagnosed correctly, that AI was the best intervention, that the workflow improved, or that the organization obtained value.

Strong technical execution applied to the wrong problem can still produce a failed investment. The system can be reliable, polished, and well evaluated at the component level while leaving the operational constraint untouched.

The broader capability is the ability to move through a complete chain of reasoning:

symptom
→ evidence
→ diagnosis
→ intervention options
→ decision
→ implementation
→ evaluation
→ operational outcome
→ business outcome

Every arrow is a claim that can fail.

The symptom may not identify the cause. The evidence may be incomplete. The chosen intervention may not affect the diagnosed mechanism. The implementation may not behave as intended. The changed workflow may not create a business result.

That is why the work begins before architecture and continues after deployment.

It requires being willing to conclude that the right intervention is process change, knowledge improvement, integration, deterministic software, analytics, or retrieval.

And when the evidence does justify AI, it gives the engineering team something much more useful than a mandate to build.

It gives them a diagnosed mechanism, an operational hypothesis, a place in the workflow, explicit alternatives, measurable outcomes, and a reason for the technology to be there.

Related engineering notes